VNS User Guide

Release 5.3.2

3HE13923AAJA

August 05, 2018

 CONTENTS

1 About this Document 2

1.1 Validity of this Document . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Technical Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2 Virtualized Network Services 4

2.1 New Features Reference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.1.1 New features for Nuage VNS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.1.2 New features in Release 5.3.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
New features for NSG hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
New features for port address translation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
New features for visualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
New features for L2 services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
New features for QoS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
New features for IPv6 (Beta) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
New features for network acceleration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
New features for SaaS routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
New features for operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
New features for NSG platform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.2 VNS overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.2.1 Nuage VNS Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.2.2 Nuage VNS Feature Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.2.3 Infrastructure Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Certificate Authority (CA) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
Key Server (KS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
Utility Host . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
VNS IP Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
NTP Infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Domain Name System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.2.4 Virtualized Services Directory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
VSD Role in VNS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.2.5 Virtualized Services Controller . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
VSC Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

3 Operations 17
3.1 Getting Started with VNS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.1.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.1.2 User Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
User Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
3.1.3 User Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
3.1.4 Infrastructure Profiles in VNS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

i
 3.1.5 VNS Provisioning Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
3.1.6 Detailed Provisioning Workflows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
1. Infrastructure Provisioning (CSPRoot Workflow) . . . . . . . . . . . . . . . . . . . . . . . 21
2. Infrastructure Provisioning (Enterprise Admin Workflow) . . . . . . . . . . . . . . . . . . 50
3. Infrastructure Provisioning (Installer Workflow) . . . . . . . . . . . . . . . . . . . . . . . 65
4. Basic Service Configuration (Network Designer Workflow) . . . . . . . . . . . . . . . . . 65
5. Advanced Service Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
3.2 Bootstrapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
3.2.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
3.2.2 Pre-bootstrap NSG image upgrades . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
Performing a pre-bootstrap NSG image upgrade . . . . . . . . . . . . . . . . . . . . . . . . . 73
Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
3.2.3 Auto-Bootstrapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
LED Indicators during Auto-Bootstrapping . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
Considerations for bootstrapping over LTE uplink . . . . . . . . . . . . . . . . . . . . . . . . 76
3.2.4 One/Two-Factor Bootstrapping (Installer Workflow) . . . . . . . . . . . . . . . . . . . . . . 77
Prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
WAN/LAN and Installer Ports on NSG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
Installer Instructions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
Viewing LTE information via Pre-bootstrap Status Page . . . . . . . . . . . . . . . . . . . . 84
Manual Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
3.2.5 CLI Based Bootstrapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
CLI Bootstrapping Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
3.2.6 Pre-Bootstrap: Status Webpage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
3.2.7 Pre-Bootstrap: Enable SSH . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
Enabling SSH Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
3.2.8 Pre-Bootstrap: NSG WAN Checks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
Sample uplink configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
3.2.9 Pre-Bootstrap: NSG info . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
3.2.10 Pre-Bootstrap and Post-Bootstrap: NSG log collection . . . . . . . . . . . . . . . . . . . . 95
3.2.11 VSC CLI Show Command Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
3.3 Gateway Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
3.3.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
3.3.2 NSG License Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
3.3.3 NSG System Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
3.3.4 Configuration Updates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
3.3.5 Configuration Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
3.3.6 Proxy Reachability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
3.3.7 NSG Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
Auto-Deactivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
Controllerless Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
Alarms for Gateway Operational States . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
3.3.8 On-Demand Gateway Deactivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
On-Demand Gateway Deactivation via CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
3.3.9 CLI-Based NSG Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
NSG image upgrade (pre-bootstrap) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
Gateway Deactivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
Certificate Renewal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
NSG Patch List . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
Export core dumps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
3.3.10 Setting up Secure Access to NSG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
3.3.11 Trusted Platform Module (TPM) Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
3.3.12 NSG Upgrade Profiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
NSG Upgrade Profiles Workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113

ii
 3.3.13 NSG Patching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
NSG Patch Profile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
NSG Patch Bundle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
Detailed Patch Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
NSG Patch List . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
NSG Patch Profiles Workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
3.3.14 Patch Operation Status Codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
Unknown Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
NSG Upgrade Statuses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
NSG Patch Statuses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
3.3.15 Datapath CLI commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
nuage-flow-top . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
ovs-appctl bridge/dump-conntracks-summary alubr0 . . . . . . . . . . . . . . . . . . . . . . 120
ovs-appctl bridge/dump-conntracks alubr0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
conntrack -L . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
3.3.16 CLI Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122

4 Network Infrastructure 129

4.1 NAT-T . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
4.1.1 NAT-T overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
Supported NAT-T features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
4.1.2 Supported NAT types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
NAT-T compatibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
4.1.3 NAT-T use cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
Dual uplink support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
VSC NAT-T mapping use cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
4.1.4 NAT-T NPM probes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
4.1.5 UBR-only NAT probes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
4.1.6 NAT-T visualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
4.1.7 NAT-T configuration specifics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
4.1.8 NAT-T workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
4.1.9 Procedures for NAT-T . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
Enabling or disabling NAT-T NPM probes in the domain . . . . . . . . . . . . . . . . . . . . 143
Configuring NAT-T settings at the NSG Port level . . . . . . . . . . . . . . . . . . . . . . . . 143
Configuring the NAT probe interval . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
4.1.10 NAT-T troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
4.1.11 NAT-T CLI samples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
nat-traversal-mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
bgp-NAT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
natt/show . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
dump Nuage_Aar_Npmg . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
analyzer/dump . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
generator/dump . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
neighbor/show . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
dump Nuage_Aar_Xpmg_Dpid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
get Static_Config_Neighbor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
get Local . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
dpif/show . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
ipsec/list-policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
4.2 Secondary IP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
4.2.1 Secondary IP overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
4.2.2 Secondary IP use case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
4.2.3 Secondary IP configuration specifics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
General considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156

iii
 BGP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
Uplink connection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
Bootstrapping considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
4.2.4 Secondary IP workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
4.2.5 Procedures for secondary IP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
Configuring a BGP neighbor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
Configuring an uplink connection for secondary IP . . . . . . . . . . . . . . . . . . . . . . . 158
4.2.6 BGP routing policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
Default BGP export routing policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
Sample BGP export routing policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
4.2.7 Secondary IP CLI samples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
ip address show port . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
nuage-bgp-show summary-all . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
nuage-bgp-show neighbor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
nuage-bgp-show routes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
nuage-bgp-show bgp-config . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
nuage-bgp-show rtm-routes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
4.3 NSG Underlay Border Router . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
4.3.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
4.3.2 Configuration Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
4.3.3 Routing via NSG-UBR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
Forwarding Behavior Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
Advanced ACL Uplink Preference Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . 169
Underlay Reachability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
Service Chaining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
4.3.4 NSG-UBR Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
Define Underlays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
Define Performance Monitor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
Instantiate NSG-UBR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
Add and/or Extend VLANs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
Define Uplink Connections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
Configure BFD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
Create NSG-UBR Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
Create NSG Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
Create NSG Group to UBR Group Bindings . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
Activate the NSG-UBR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174
4.3.5 Border Router Support on NSG-UBR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174
4.3.6 Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
4.3.7 Sample BFD Peer Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
7750 SR Sample . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
Cisco Sample . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
4.3.8 CLI Show Command Sample . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176
BFD CLI commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176
UBR routes CLI command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
4.4 NSG Border Router . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
4.4.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
4.4.2 Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
Permission Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
4.4.3 Controller Topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
4.4.4 NSG-BR Redundancy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
4.4.5 Multi-tenancy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
IPsec Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
4.4.6 BR Backbone . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
4.4.7 Tunnel Shaping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183

iv
 4.4.8 Underlay Reachability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
4.4.9 NSG-BR Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
CSPRoot Workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
Instantiate NSG-BR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184
Bootstrap the NSG-BR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184
Assign Permission . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184
Create BR Connection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
Configure BFD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
Enterprise Admin Workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
Link Domains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
Configure Demarcation Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
Define End-to-End ACLs across Linked Domains . . . . . . . . . . . . . . . . . . . . . . . . 192
4.4.10 BR Backbone Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
Add Static Route in DC Domain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
Configure Demarcation Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
4.4.11 Tunnel Shaping Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
Define Tunnel Shaping Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
Define Rate Limiter(s) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
Create Tunnel Shaper QoS Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
Assign Policy to Uplink VLAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
Configuration Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
4.4.12 Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
4.4.13 Sample CLI output for Tunnel Shapers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
4.4.14 Sample CLI Output for BFD Sessions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
4.5 Network Uplinks (Wired and LTE) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
4.5.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202
4.5.2 LTE Uplink . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202
4.5.3 Use Case Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202
4.5.4 Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
Heterogeneous transport network - Tunnel (VxLAN/IPSec) . . . . . . . . . . . . . . . . . . . 203
Heterogeneous transport network - no Tunnel . . . . . . . . . . . . . . . . . . . . . . . . . . 203
4.5.5 LTE Staging and Operations Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . 204
Verifying port mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
Changing SIMs for integrated LTE devices . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
4.5.6 Provisioning Overview (Wired and LTE Uplinks) . . . . . . . . . . . . . . . . . . . . . . . 206
LTE Specific Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
4.5.7 Uplink Role . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
4.5.8 Configuring Controller(s) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208
4.5.9 Forwarding Behavior . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
Uplink Preference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
4.5.10 Network Acceleration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212
Network Acceleration Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212
Network Acceleration Verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
4.5.11 NSG Controller Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214
4.5.12 Controller Resiliency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215
Scenario 1: NSG uplink(s) not behind a NAT device . . . . . . . . . . . . . . . . . . . . . . 215
Scenario 2: NSG uplink is behind NAT device . . . . . . . . . . . . . . . . . . . . . . . . . 216
4.5.13 IPv6 uplinks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
Feature support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
4.5.14 Uplink Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218
NSG Uplink Statistics Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
NSG-UBR Uplink Statistics Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
LTE Uplink Statistics, Status and Dongle Information . . . . . . . . . . . . . . . . . . . . . . 222
4.5.15 CLI Samples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226

v
 Network Acceleration CLI Samples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228

5 Network Services 236

5.1 DHCP on Bridge Ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
5.1.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
DHCP Decline Handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238
5.1.2 Provisioning Workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238
Define DHCP Pool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238
Modifying DHCP Address Ranges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240
DHCP Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
5.1.3 CLI Show Command Sample . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
5.2 Access Resiliency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242
5.2.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242
5.2.2 Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242
5.2.3 Feature Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
RG Creation Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
5.2.4 Failure scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
Direct physical port failure on master NSG . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
Direct physical single LAN switch failure . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
Uplink failure/OF-TLS session loss on Master NSG . . . . . . . . . . . . . . . . . . . . . . 245
Failure in NSGs with dual uplinks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
5.2.5 Provisioning Workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
Create New Redundant Group (RG) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
Add RG Ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
Add new VLANs to RG Ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249
Add vPorts on RG Port . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252
5.2.6 Interworking with other NSG features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
Egress/Ingress QoS: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
Port Address Translation: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
5.2.7 CLI Show Command Sample . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254
5.3 Port Address Translation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
5.3.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
5.3.2 Use Case Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257
5.3.3 NSG Forwarding Behavior . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258
5.3.4 Per Uplink NAT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259
5.3.5 Overlapping PAT Pools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259
5.3.6 Address Pool Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259
5.3.7 Provisioning Workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260
Enabling PAT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260
Configuring Address Translation Pools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262
Advertising Address Pools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266
5.3.8 PAT Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
Displaying PAT Statistics on VSD UI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
5.3.9 CLI Show Command Sample . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
5.4 PAT to Overlay . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
5.4.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
5.4.2 Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278
5.4.3 Routing Policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278
Configuration Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
Preference Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
5.4.4 PAT to Overlay Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
Link Domains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280
Create Address Translation Pools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282
Define 1:1 NAT Maps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283

vi
 Add Static Route . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
Configure a Routing Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284
5.4.5 CLI Show Command Sample . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285
5.5 Bi-Directional NAT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287
5.5.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287
Provider to Customer SPAT/SNAT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287
Provider to Customer DNAT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288
Customer to Provider SPAT/SNAT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289
5.5.2 Configuration Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 290
Provider Domain Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 290
Customer Domain Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 290
Domain Links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 290
5.5.3 Bi-Directional NAT Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 290
5.5.4 Provisioning Workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291
Specify SPAT Source Pool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291
Create Bi-Directional Link . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292
Provider to Customer Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292
Customer to Provider Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293
5.5.5 Sample CLI Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293
5.6 Service Chaining in VNS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295
5.6.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295
1. RT in DC domain via NSG-BR using BR connection . . . . . . . . . . . . . . . . . . . . . 295
2. RT on Branch vPort in a disjoint underlay via NSG-UBR . . . . . . . . . . . . . . . . . . 296
3. RT on NSG-BR/NSG-UBR vPort in common underlay . . . . . . . . . . . . . . . . . . . 296
5.6.2 Service Chaining Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296
Define Redirect Target (RT) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297
Associate RT to a vPort . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297
Define Forwarding Policy in Branch Domain . . . . . . . . . . . . . . . . . . . . . . . . . . 298
5.6.3 Sample CLI Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298
Example Output for Use Case 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298
Example Output for Use Case 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301
5.7 DNA Subnets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303
5.7.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303
Proxy ARP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303
5.7.2 Sample Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304
Sample #1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304
Sample #2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304
5.7.3 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305
Disable Advertising on a Subnet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305
Configure a Proxy ARP Filter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305
5.7.4 CLI Samples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306
5.8 QoS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307
5.8.1 QoS overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308
Supported QoS features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308
5.8.2 Ingress and egress QoS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308
Ingress and egress QoS use cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309
QoS algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311
QoS statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312
NSG-UBR QoS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312
5.8.3 Control traffic QoS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312
Control traffic QoS for the VSC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312
5.8.4 Ingress and egress QoS configuration specifics . . . . . . . . . . . . . . . . . . . . . . . . . 313
QoS policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313
DSCP mapping tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313

vii
 Ingress forwarding policies - DSCP remarking . . . . . . . . . . . . . . . . . . . . . . . . . 313
Egress QoS policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313
Rate limiters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 314
Remarking profiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 314
Ingress QoS policers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315
QoS statistics configuration specifics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315
5.8.5 QoS workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318
5.8.6 Procedures for QoS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318
Creating rate limiters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318
Creating DSCP remarking profiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318
Creating DSCP remarking profiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319
Creating QoS policers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319
Creating egress QoS policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319
Assigning QoS policies to a port/VLAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 320
Displaying egress QoS statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 320
5.8.7 Control traffic QoS procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 320
5.8.8 QoS CLI samples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321
Configuring remarking for control traffic on the VSC . . . . . . . . . . . . . . . . . . . . . . 322

6 Routing Protocols 324

6.1 BGP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325
6.1.1 BGP Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 326
MD5 Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 328
Multi-hop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 328
BGP Session Redundancy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 328
6.1.2 Default Route From LAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330
6.1.3 VIF RTM Routes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330
6.1.4 BGP Configuration Workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330
Enable BGP and configure Local Autonomous System (AS) . . . . . . . . . . . . . . . . . . 330
Define Routing Policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 332
Create BGP Profile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 334
BGP Peering with PE/Uplink . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 336
BGP Peering with CPE/Access . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 338
6.1.5 XML Configuration Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 338
6.1.6 CLI Show Command Sample . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 338
6.1.7 BGP Blob . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 339
Sample BGP Neighbor XML Blob . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 339
Sample Routing Policy XML Blob . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 341
Order of Inheritance of Default Actions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345
6.1.8 BGP Policy Routing Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345
Central Gateway . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 346
Distributed Interworking — LAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 351
Distributed Interworking — WAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 352
6.1.9 BGP CLI Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353
6.1.10 VIF RTM CLI Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 377
6.2 OSPF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 379
6.2.1 OSPF Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 379
OSPF Areas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 380
6.2.2 OSPF Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 381
OSPF Instance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 381
OSPF Area . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 382
OSPF Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383
Routing Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383
Policy Object Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 384

viii
 NSG Routing Policy Binding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 384
6.2.3 Provisioning Workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 385
Enable Routing and Configure an AS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 385
Create and Instantiate an L3 Domain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 385
Create Routing Policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 386
Create an OSPF Instance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 386
Create an OSPF Area . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 386
Create an OSPF Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 387
Create Policy Object Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 387
Create NSG Routing Policy Bindings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 387
6.2.4 OSPF Routing Policy Blob . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 388
OSPF Routing Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 389
6.2.5 OSPF CLI Samples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 389

7 Encryption 402
7.1 Group-Key IPsec . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403
7.1.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403
7.1.2 Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 404
7.1.3 Use Case: VXLANoIPsec . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 405
7.1.4 IPsec Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 406
7.1.5 Overview of Cryptographic Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 407
SEK Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 407
Seed Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 407
TEK Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 408
Processing of Forced Rekey and Revoke Events . . . . . . . . . . . . . . . . . . . . . . . . . 408
Processing for Multi-tenant NSG-BR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 409
Processing during Controllerless Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 409
7.1.6 Workflows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 409
High-Level Feature Workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 409
Detailed Provisioning Workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 410
7.1.7 VxLANoIPsec QoS Marking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 416
VxLAN QoS Marking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 416
7.1.8 Interworking with NAT-T . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 419
7.1.9 CLI Command Sample Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 419
7.2 IKE IPsec . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 421
7.2.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 421
7.2.2 Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423
7.2.3 Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423
Feature Interworking Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423
7.2.4 Architecture Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 424
Forwarding Behavior . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 424
IKE Protocol Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 425
7.2.5 HTTP Ping for IKE Tunnels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 425
7.2.6 IKE Security Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 425
High Level Workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 425
Detailed Workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 427
7.2.7 IKE Connection Redundancy in 1:Many Topology . . . . . . . . . . . . . . . . . . . . . . 437
7.2.8 CLI Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 438

8 Application-Aware Routing 445

8.1 AAR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 446
8.1.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 447
Application Discovery (AD) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 447
Network Performance Measurement (NPM) . . . . . . . . . . . . . . . . . . . . . . . . . . . 447

ix
 Application Policy and Visualization (APV) . . . . . . . . . . . . . . . . . . . . . . . . . . . 447
8.1.2 Configuration Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 448
Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 448
APM Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 448
Performance Monitors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 449
Performance Monitor Scopes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 449
NPM Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 449
8.1.3 SLA-Aware Path Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 449
8.1.4 AAR for NSG-UBR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 451
8.1.5 Feature Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 451
8.1.6 Configuration Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 452
Enable Performance Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 452
Enable AD and APV . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 453
Define Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 454
Define Performance Monitor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 458
Create APM Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 458
Create Application Binding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 459
Associate APM Group with Domain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 460
Define NPM Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 461
8.1.7 Provisioning Workflows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 464
AD and APV Workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 464
NPM Workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 465
8.1.8 CLI Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 465
8.1.9 Performance Monitoring Debug CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 468
8.1.10 DPI Debug CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 503
8.2 AAR Visualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 505
8.2.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 507
8.2.2 Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 508
8.2.3 Workflows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 508
Access AAR Visualization Reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 508
Monitor and analyze application usage at the enterprise level . . . . . . . . . . . . . . . . . . 509
Monitor and analyze application usage at the domain level . . . . . . . . . . . . . . . . . . . 509
8.2.4 Scrolling Graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 510
8.2.5 Enterprise Dashboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 511
SLA Status - Flow effectiveness score . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 511
Top 5 Discovered Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 511
Top 5 Upload Users . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 511
Top 5 Download Users . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 511
Newly Discovered Default Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 512
8.2.6 Enterprise Detail Dashboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 512
Enterprise Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 512
Data Usage on Apps per NSG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 512
Data Usage per NSG for Selected Application . . . . . . . . . . . . . . . . . . . . . . . . . . 512
Cumulative Data Usage on NSG for Selected Application . . . . . . . . . . . . . . . . . . . 513
NSG Application Traffic Details for Source IP . . . . . . . . . . . . . . . . . . . . . . . . . 513
Enterprise - Top 20 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 513
Application specific Date Histogram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 513
8.2.7 Enterprise Map View . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 514
Map View Prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 514
Map View Dashboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 515
8.2.8 NSG Dashboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 518
NSG Application Traffic Detail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 518
NSG Application Traffic Details for Application . . . . . . . . . . . . . . . . . . . . . . . . 518
NSG Application Traffic Details for Source IP . . . . . . . . . . . . . . . . . . . . . . . . . 518

x
 Per Port Traffic Visualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 519
Top 10 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 519
Top 5 Discovered Application - Usage over Time . . . . . . . . . . . . . . . . . . . . . . . . 519
Top 5 Talkers (Upload) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 519
Top 5 Talkers (Download) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 519
Alarms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 519
8.2.9 NSG Detail Dashboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 520
From NSG Application Report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 520
To NSG Application Report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 520
From NSG SLA Report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 520
To NSG SLA Report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 520
8.2.10 NAT-T Events Dashboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 521
8.2.11 IKE Dashboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 521
8.2.12 Alarms Dashboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 521
8.2.13 Domain - Applications Dashboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 521
Top 20 Talkers in Domain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 521
Top 5 Applications in Domain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 521
Top 5 Application Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 522
8.2.14 Domain - Network Performance Dashboard . . . . . . . . . . . . . . . . . . . . . . . . . . 522
From NSG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 522
To NSG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 522
Probe Detail (NSG as Source) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 522
Probe Detail (NSG as Destination) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 523
8.2.15 Domain - Application Performance Management Dashboard . . . . . . . . . . . . . . . . . 523
From NSG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 524
To NSG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 524
SLA HeatMap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 524
Traffic Uplink Pairs for Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 524
Traffic for Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 525
SLA Packet Loss for Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 525
SLA Delay for Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 525
SLA Jitter for Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 525
8.2.16 Domain - Details Dashboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 525
Domain Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 525
Data Usage on Apps per NSG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 526
Data Usage per NSG for Selected Application . . . . . . . . . . . . . . . . . . . . . . . . . . 526
Cumulative Data Usage on NSG for Selected Application . . . . . . . . . . . . . . . . . . . 526
NSG Application Traffic Details for Source IP . . . . . . . . . . . . . . . . . . . . . . . . . 526
8.2.17 Query Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 526
Enterprise Dashboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 526
Enterprise Detail Dashboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 532
NSG Dashboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 535
NSG Detail Dashboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 541
Domain - Applications Dashboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 553
Domain - Network Performance Dashboard . . . . . . . . . . . . . . . . . . . . . . . . . . . 559
8.3 SaaS Routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 561
8.3.1 SaaS routing overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 561
Supported SaaS routing features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 561
8.3.2 SaaS application discovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 562
8.3.3 SaaS performance and reachability monitoring . . . . . . . . . . . . . . . . . . . . . . . . 562
HTTP ping performance monitoring with CLI . . . . . . . . . . . . . . . . . . . . . . . . . . 563
HTTP Ping Visualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 564
8.3.4 SaaS path selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 566
Per uplink NAT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 566

xi
 Per uplink breakout preference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 567
8.3.5 SaaS routing configuration specifics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 567
SaaS application types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 567
SaaS application groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 568
HTTP performance monitor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 568
Forwarding path lists . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 569
8.3.6 Procedures for SaaS routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 570
Configuring SaaS application types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 570
Configuring SaaS application groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 570
Creating a Performance Monitor for HTTP Ping . . . . . . . . . . . . . . . . . . . . . . . . 570
Assigning an HTTP performance monitor to an IKE tunnel . . . . . . . . . . . . . . . . . . . 571
Configuring forwarding path lists . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 571
8.3.7 Predefined SaaS application type updates . . . . . . . . . . . . . . . . . . . . . . . . . . . 572
Predefined SaaS application type update sample . . . . . . . . . . . . . . . . . . . . . . . . . 572
8.3.8 SaaS routing CLI samples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 572

9 Branch Services 579

9.1 Virtual Network Function on NSG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 580
9.1.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 581
Supported VNF features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 582
VNF onboarding and bootstrapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 584
Datapath integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 584
EMS management of VNFs on NSG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 585
Feature support considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 585
9.1.2 VNF life-cycle states and actions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 586
States . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 587
Actions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 588
9.1.3 Configuration specifics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 588
Enabling VNF management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 588
Storing and configuring input files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 589
VSD objects and policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 589
9.1.4 Workflows for VNF on NSG features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 591
Firewall workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 591
WAN-Optimization workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 591
9.1.5 Procedure for VNF Onboarding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 592
Prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 592
Creating the Bootstrap ISO File for the VNF . . . . . . . . . . . . . . . . . . . . . . . . . . 592
9.1.6 VSD procedures for VNF on NSG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 593
Creating the VNF Metadata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 593
Creating the VNF Descriptor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 594
Creating the VNF Interface Descriptors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 596
Creating a VNF Instance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 597
Redirecting Traffic to the VNF (firewall) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 603
Redirecting Traffic to the VNF (WAN-Optimization) . . . . . . . . . . . . . . . . . . . . . . 606
9.1.7 Procedures for WAN-Optimization with Ipanema . . . . . . . . . . . . . . . . . . . . . . . 608
Verifying the NSG partition configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 608
Disable DWS feature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 609
Define default ingress and egress ACL rules on the VNF . . . . . . . . . . . . . . . . . . . . 609
Viewing the required IP addresses in VSD . . . . . . . . . . . . . . . . . . . . . . . . . . . . 609
Ipanema system provisioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 610
Ipanema application group provisioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 612
Enable compression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 612
Verify compression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 612
9.1.8 Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 612

xii
 CPU cycle consumption due to ARP broadcast packets . . . . . . . . . . . . . . . . . . . . . 612
Error States . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 613
9.1.9 VNF metadata XML sample blobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 613
Firewall XML blob . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 613
WAN-Optimization XML blob . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 614
9.1.10 CLIs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 615
9.2 WiFi on NSG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 616
9.2.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 616
9.2.2 Provisioning Workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 617
Create WiFi Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 617
Create Captive Portal Profile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 619
Edit/Create SSID Connection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 621
Attach Host/Bridge vPort . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 623
Define DHCP Pool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 623
End User Workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 624
9.2.3 AP Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 625
9.2.4 Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 625
9.2.5 Sample CLI Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 626

10 Resources 629
10.1 VSC Tools Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 630
10.1.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 631
10.1.2 Show Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 631
Help . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 631
OF-Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 632
Agent-Version . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 633
Agent-Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 634
Bridge-Flows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 634
Service-Flows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 635
Service-Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 637
Service-MAC-Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 640
FIB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 641
Ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 642
Port-Stats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 643
Service-Subnets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 644
Service-Domains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 645
Datapath-Flows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 646
Datapath-Ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 647
Tunnel-Info . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 647
ACL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 648
System-Info . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 650
QoS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 651
Port-Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 651
System-Connections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 651
NTPQ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 652
IP-Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 652
IP-Table-NAT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 653
Routing-Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 654
10.1.3 Debug Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 654
Help . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 654
Tcpdump . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 655
Ethtool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 656
Dig . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 656
10.1.4 Configure Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 657

xiii
Help . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 657
Port-Up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 658
Port-Down . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 658
Reboot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 658
Shutdown . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 659

xiv
 VNS User Guide, Release 5.3.2

Release: 5.3.2
Issue: 1
Issue Date: August 05, 2018
Document Number: 3HE13923AAJA

NUAGE NETWORKS – PROPRIETARY & CONFIDENTIAL

This document contains proprietary/trade secret information which is the property of Nokia Corporation. Not to be
made available to, or copied or used by anyone who is not an employee of Nokia Corporation except when there is a
valid non-disclosure agreement in place which covers such information and contains appropriate non-disclosure and
limited use obligations.
This document is protected by copyright. Except as specifically permitted herein, no portion of the provided infor-
mation can be reproduced in any form, or by any means, without prior written permission from Nokia Corporation /
Nuage Networks.
Nuage Networks and the Nuage Networks logo are trademarks of the Nokia group of companies. Nokia is a registered
trademark of Nokia Corporation. Other product and company names mentioned herein may be trademarks or trade
names of their respective owners.
The information presented is subject to change without notice.
Nokia Corporation / Nuage Networks assumes no responsibility for inaccuracies contained herein.

Copyright©2018 Nokia Corporation / Nuage Networks. All rights reserved.

Build Number: 31

CONTENTS 1
 CHAPTER

ONE

ABOUT THIS DOCUMENT

• Validity of this Document (page 3)

• Audience (page 3)
• Technical Support (page 3)

2
 VNS User Guide, Release 5.3.2

For a complete list of applicable user documentation, see the Technical Publications section of the Release Notes for
your Nuage Networks software version.

1.1 Validity of this Document

Printed versions of this document may not be up to date. Only the Web version of this document is current.

1.2 Audience

This manual is intended for enterprise system administrators who are responsible for enterprise network configuration
and administrators for the Nuage VSP/VNS software. It is assumed that the reader is familiar with virtualization and
networking technologies. Other assumptions are explicitly called out in the relevant chapters.

1.3 Technical Support

If you purchased a service agreement for your Nuage Networks VSP/VNS solution and related products from a dis-
tributor or authorized reseller, contact the technical support staff for that distributor or reseller for assistance. If you
purchased an Alcatel-Lucent or Nokia service agreement, contact your welcome center:
https://networks.nokia.com/support
Nokia Online Services (NOLCS) provides registered customers with access to technical support, software downloads,
training, documentation, literature, and other related assets for our products and solutions. For assistance with NOLCS,
including inability to access, contact us as follows:
• Inside the U.S. and Canada: 1-866-582-3688, prompt 7.
• Outside the U.S.: 1-630-224-9000
• Via email: NOLS.support@nokia.com

1.1. Validity of this Document 3

 CHAPTER

TWO

VIRTUALIZED NETWORK SERVICES

• New Features Reference (page 5)

• VNS overview (page 10)

4
 VNS User Guide, Release 5.3.2

2.1 New Features Reference

• New features for Nuage VNS (page 6)

• New features in Release 5.3.2 (page 6)
– New features for NSG hardware (page 6)

* 7850 NSG-E202 with integrated WiFi (page 6)

* 7850 NSG-E206/306 with integrated WiFi and LTE (page 6)
– New features for port address translation (page 6)

* Overlapping PAT pools (page 6)

– New features for visualization (page 6)

* Enterprise map view (page 6)

– New features for L2 services (page 7)

* Application discovery for L2 services (page 7)

* L7 ACL support for DSCP rewrite (page 7)
– New features for QoS (page 7)

* Ingress ACL support for DSCP remarking of customer packets (page 7)

* QoS for NSG-UBR (page 7)
– New features for IPv6 (Beta) (page 8)

* IPv6 network uplinks (Beta) (page 8)

– New features for network acceleration (page 8)

* Network acceleration for NSG-X200, NSG-E300, NSG-E200, NSG-E, and NSG-C (page 8)
* Network acceleration for NSG-V (page 8)
– New features for SaaS routing (page 8)

* SaaS application discovery — first packet detection (page 8)

* SaaS application path selection — per uplink breakout preference (page 8)
– New features for operations (page 9)

* Pre-bootstrap CLI-based image upgrade (page 9)

* Grant nuage user permission to transfer coredumps (page 9)
* Persist logs across NSG reboot and power down events (page 9)
* Command to display NAT namespace rules for troubleshooting NAT features (page 9)
* VSD API access restrictions for NSG (page 9)
– New features for NSG platform (page 9)

* Integrated LTE platform support (NSG-E206 and NSG-E306) (page 9)

2.1. New Features Reference 5

 VNS User Guide, Release 5.3.2

2.1.1 New features for Nuage VNS

This chapter highlights new features for Nuage VNS and provides pointers into the documentation for information
about using the features. Feature lists and highlevel feature descriptions are also available in the VNS Release Notes.
Some releases may not be listed in this chapter, either because no new features are introduced or the features introduced
do not require documentation.

2.1.2 New features in Release 5.3.2

This section lists the features introduced in Release 5.3.2.

New features for NSG hardware

7850 NSG-E202 with integrated WiFi

The new hardware model NSG-E202 is a variant on the existing NSG-E200 model. It has the same front and rear
panels as the NSG-E200, but also includes integrated WiFi antennas. This new model has access to the Nuage VNS
WiFi feature set without the use of an added WiFi dongle.
Refer to the NSG Bootstrapping Webpage for more information about the 7850 NSG-E202:
http://bootstrap.nuagenetworks.net/nsg/

7850 NSG-E206/306 with integrated WiFi and LTE

The new hardware models NSG-E202 and NSG-306 are variants on the existing NSG-E200 and NSG-E300 models.
They have the same front and rear panels as the NSG-E200/300, but also include integrated WiFi and LTE antennas.
These new models have access to the Nuage VNS WiFi and LTE feature sets without the use of added WiFi or LTE
dongles.
Refer to the NSG Bootstrapping Webpage for more information about the 7850 NSG-E206/306:
http://bootstrap.nuagenetworks.net/nsg/

New features for port address translation

Overlapping PAT pools

You can now define NSG PAT pools with overlapping IP address ranges. This allows you to use 1:1 or N:1 address
maps when the PAT pool contains private IP addresses that recur across multiple NSGs.
• Port Address Translation (page 255)
• Overlapping PAT Pools (page 259)

New features for visualization

Enterprise map view

The visualization feature set has been enhanced with the addition of the enterprise map view. The enterprise map view
provides a visualization of the geographic location and administrative status of all NSGs in the enterprise. The map

2.1. New Features Reference 6

 VNS User Guide, Release 5.3.2

view uses Google Maps and user-specified NSG locations to map branch locations on an interactive, navigable map.
Using the map view, you can search and navigate to NSGs by region and quickly determine which NSGs are in a
pre-bootstrap state or require a software upgrade.
In addition, the NSG dashboard has been enhanced with the an alarms summary. The alarms summary shows how
many alarms are raised on the NSG, and also allows you to navigate to the alarms dashboard. The alarms dashboard
provides a summary of information for all alarms currently raised on the NSG.
• Enterprise Map View (page 514)
• Alarms Dashboard (page 521)

New features for L2 services

Application discovery for L2 services

The application discovery (AD) feature set has been extended to support services in L2 domains. You can use AD to
identify and monitor incoming traffic using L7 application classification with a library of application signatures. You
can also create custom classification rules based on IP address, L4 ports, or L4 protocols.
• AAR (page 446)

L7 ACL support for DSCP rewrite

The layer 7 access control list (L7 ACL) feature set has been extended to support services in L2 domains. You can
use Advanced Ingress Forwarding ACLs based on L7 application signatures, allowing you to map detected application
traffic to a specific forwarding class.
• “Security Policy: Layer 7 ACLs” chapter in the VSP User Guide

New features for QoS

Ingress ACL support for DSCP remarking of customer packets

You can now configure DSCP remarking of the inner customer packets using an ingress forwarding policy. Customer
packet DSCP remarking allows you to redefine the forwarding class assigned to the ingress customer payload packet.
• Ingress and egress QoS (page 308)
• Ingress forwarding policies - DSCP remarking (page 313)

QoS for NSG-UBR

Egress QoS functionality has been extended to the NSG-UBR for a single-tenant use case only. This means that the
ability to prioritize traffic between two different customers is not supported. If traffic from multiple customers is routed
through the NSG-UBR, all priority queues are treated equally and all WRR queues are part of the same hierarchy.
• QoS (page 307)

2.1. New Features Reference 7

 VNS User Guide, Release 5.3.2

New features for IPv6 (Beta)

IPv6 network uplinks (Beta)

Support for IPv6 has been introduced at beta quality for static and dynamic NSG network uplinks. Full IPv6 support
for VNS features is planned to be introduced over multiple releases.
• IPv6 uplinks (page 217)
• Define Uplink Connections (page 54)

New features for network acceleration

Network acceleration for NSG-X200, NSG-E300, NSG-E200, NSG-E, and NSG-C

In previous releases, network acceleration has been supported on the NSG-X only. In release 5.3.2, network accel-
eration support has been extended to the NSG-X200, NSG-E300, NSG-E200, NSG-E, and NSG-C. This includes
throughput acceleration for overlay only data paths with DPI.
• Network Acceleration (page 212)
• Network Acceleration CLI Samples (page 228)

Network acceleration for NSG-V

In previous releases, network acceleration has been supported on physical NSGs only. In release 5.3.2, network
acceleration support has been extended to the NSG-V. This support is functional only.
• Network Acceleration (page 212)
• Network Acceleration CLI Samples (page 228)

New features for SaaS routing

SaaS application discovery — first packet detection

SaaS applications can now be discovered using a database of network macros to identify SaaS IP addresses at the
first packet. This discovery mechanism ensures that the traffic flow does not switch paths after the application is
discovered. The database includes network macros for eight predefined SaaS application types, but you can also
create custom SaaS application types with associated network macros.
• SaaS application discovery (page 562)
• SaaS application types (page 567)
• SaaS application groups (page 568)
• Procedures for SaaS routing (page 570)

SaaS application path selection — per uplink breakout preference

You can now use forwarding path lists to assign an uplink preference for different breakout types in a dual uplink
scenario. You can specify that traffic be redirected through an IKE tunnel, or through PAT to underlay and route to
underlay. This mechanism allows the NSG to have one MPLS link and one Internet link.

2.1. New Features Reference 8

 VNS User Guide, Release 5.3.2

• Per uplink breakout preference (page 567)

• Forwarding path lists (page 569)
• Procedures for SaaS routing (page 570)

New features for operations

Pre-bootstrap CLI-based image upgrade

You can now perform CLI-based image upgrades for pre-bootstrap NSGs.
• Pre-bootstrap NSG image upgrades (page 73)

Grant nuage user permission to transfer coredumps

The nuage user can now run a command to transfer coredumps to a readable location.
• Export core dumps (page 108)

Persist logs across NSG reboot and power down events

The NSG now copies the logexport.tar.gz file to /home/user/nuage when the NSG undergoes a graceful reboot or
shutdown.
• Pre-Bootstrap and Post-Bootstrap: NSG log collection (page 95)

Command to display NAT namespace rules for troubleshooting NAT features

The following commands are now whitelisted for the nuage user on the NSG:
• iptables -t nat -L
• iptables -t nat -L -n

VSD API access restrictions for NSG

The API authentication model is now enhanced so that the NSG can only query the API call for its own NSG instance
in the VSD. It cannot query any other NSGs.

New features for NSG platform

Integrated LTE platform support (NSG-E206 and NSG-E306)

Integrated LTE NSG platforms NSG-E206 and NSG-E306 are now supported. The integrated LTE NSG platforms
provide the same feature parity that Nuage supported with LTE dongles in the previous releases, including bootstrap
over LTE and Aux port functionality where LTE can act as a backup uplink for a wired uplink link
• LTE uplink (page 202)

2.1. New Features Reference 9

 VNS User Guide, Release 5.3.2

2.2 VNS overview

• Nuage VNS Overview (page 10)

• Nuage VNS Feature Overview (page 12)
• Infrastructure Requirements (page 12)
– Certificate Authority (CA) (page 13)
– Key Server (KS) (page 13)
– Utility Host (page 13)
– VNS IP Network (page 13)
– NTP Infrastructure (page 14)
– Domain Name System (page 14)
• Virtualized Services Directory (page 14)
– VSD Role in VNS (page 14)
• Virtualized Services Controller (page 15)
– VSC Protocols (page 15)

* OF-TLS (page 15)

* XMPP (page 15)
* JSON-RPC (page 15)
* BGP/IGP (page 15)
* MP-BGP (page 16)

2.2.1 Nuage VNS Overview

Nuage Networks Virtualized Network Services (VNS) is a new wide area network service construct enabling delivery
of business connectivity services to branch locations regardless of size or geography. The network service is indepen-
dent of the transport utilized, which provides maximum flexibility in terms of service reach and access technology.
VNS provides SDN-enabled networking with support for Layer 2 to Layer 4 services, and allows advanced network
functions such as firewalls to be deployed as part of the core service. VNS supports an intuitive GUI for service creation
and comprehensive self-management, thus reducing the requirement for high-touch complex provisioning/engineering
teams at branch locations.
Nuage VNS may be deployed as:
• Managed Service where by a Service Provider, MSO or Datacenter operator (VNO) offers a managed service
to an enterprise user
• Internal Service where by a Service Provider, MSO or Enterprise uses VNS to better manage connectivity and
services offered to its own internal sites.
• Wholesale/retail model where by a Network Service Provider sells VNS services to the ISP’s, which then
internally sell the services to enterprises. The Service Provider, MSO and Datacenter operator offers a managed
service to the enterprise user.
At a high level, the main components in the Nuage VNS solution are:

2.2. VNS overview 10

 VNS User Guide, Release 5.3.2

• Virtualized Services Directory (page 14) (VSD)

Nuage VSD is a programmable policy and analytics engine. It provides a flexible and hierarchical
network policy framework that enables IT administrators to define and enforce resource policies
in a user-friendly manner. The VSD contains a network service directory that supports role-based
administration of network resources. It is where network configuration including moves, adds and
changes are centrally managed via an intuitive graphical user interface. The VSD can be deployed as
a stand alone or clustered solution depending on scaling needs. An abstracted REST API provides
simple methods of provisioning customer services.
From within the VSD network administrators can centrally view and change the running policies on
the network including deployment of new policies on a single, multiple or network-wide basis. The
VSD is also the point for network traffic collection where site-specific and network-wide trending
reports are available. The VSD also provides sophisticated rules for collecting information on the
status of your network service. This includes functions such as collection frequencies and rolling
averages that allow you to build comprehensive Threshold Crossing Alerts (TCA) for both current
and historic information on the service performance. Statistics are aggregated over hours, days and
months and stored in Elastic Search to facilitate data mining and performance reporting.
• Virtualized Services Controller (page 15) (VSC)
Nuage VSC is the industry’s most powerful SDN controller. It functions as the robust network control
plane for the network services, maintaining a full view of the network and service topologies
Through the VSC, virtual routing and switching constructs are established to program the network-
forwarding plane using the OpenFlow™ protocol. Multiple VSC instances can be federated within
and across the network by leveraging Multi-Protocol Border Gateway Protocol (MP-BGP) – a proven
and highly scalable network technology that allows the network service to grow with the requirements
of your business whether you are operating across the country or around the globe.
• Network Services Gateway
The Network Services Gateway (NSG) constitutes the network-forwarding plane for customers’ net-
work services at the central and remote business locations. By installing an NSG at every Enterprise
site acting as a CPE, the Nuage VNS solution provides the capability to create overlay VPNs to inter-
connect customer sites and/or data centers. Connectivity between NSG endpoints can be full mesh
or hub and spoke as required. Service chaining is instantiated through service policies defined in the
VSD.
With support for both hardware and software form factors, the NSG provides maximum flexibility
to meet the demands of a customer’s sites.The hardware-based option includes form-factors to meet
the diverse throughput, network interface and network functionality requirements of their locations.
The software image utilizes the available x86-based virtualized compute platforms customers may
have at their sites or can be run on Nuage Networks recommended common off-the-shelf x86-based
network devices procured via your own channels or directly procured by the customer.
The NSG encapsulates and de-encapsulates user traffic, enforcing Layer 2 to Layer 4 network policies
as defined by the VSD. Services can be applied to the NSGs centrally on a service-wide or location-
specific deployment model. This ensures that your network service is always configured with the
standard policies applicable to your business.
Deployment of the NSG is provided by the innovative Bootstrapping (page 71) functionality of the
Nuage Networks VNS solution. When a new NSG is connected to the network, it calls home to the
VSC and is authorized by the VSD. From there a two-step authentication process is initiated to bring
the new site on to the network service.
The automated nature of this bootstrap function reduces the requirement for specialist networking
resources at your remote locations. In most cases your branch staff can unbox and plug in the NSG
themselves, which lowers the costs of service deployment.

2.2. VNS overview 11

 VNS User Guide, Release 5.3.2

2.2.2 Nuage VNS Feature Overview

The following Nuage VNS features are described in detail in the document:
• Bootstrapping (page 71) is a key installation capability to securely commission NSGs into the VNS network.
• Application-Aware Routing (page 446) enables policy-driven application performance management in a multi-
path network. Specifically, AAR enables capabilities such as application discovery, network performance mea-
surement, and intelligent path selection for application traffic based on path performance.
• QoS (page 307) is the capability to control the rate of traffic egressing the NSG ports.
• Port Address Translation (page 255) enables internal user/application IP addresses to be mapped to a single
external routable address, as well as 1:1 NAT and 1:1PAT mapping of external user/ application IP address to
external routable address.
• PAT to Overlay (page 277) enables the NSG with a distributed destination based PAT function from a branch
domain to a local or remote destination domain.
• DHCP on Bridge Ports (page 237) is NSG’s ability to service requests from multiple unique hosts connected to
the same VPort-type bridge and allocating a unique IP address to all such host requests.
• NAT Traversal (page 130) is the capability that enables Internet connected NSGs that are front-ended by NAT
devices to be part of the VNS network. The NAT-T feature allows VNS data and control plane connections to
traverse the NAT devices.
• Access Resiliency (page 242) enables NSGs to be deployed as a redundant pair enabling HA configurations,
protecting against link and device failures.
• Dual Uplink (page 201) capability enables an NSG to have two network uplinks for data/control path redundancy
as well as to allow traffic to be routed over two transport networks (e.g. IPVPN and Internet)
• IPsec (page 403) is a security capability that allows traffic to/from NSGs to be encrypted. NSG to NSG traffic
is encrypted using group-key based IPsec. NSG to 3rd party IKE Gateways is encrypted using IKE-based IPsec
(page 421).
• NSG Border Router (page 178) enables a secure demarcation point between the WAN and the DC underlay
networks. It provides seamless connectivity between NSG and non IPsec-capable endpoints (e.g. VRS, 7750
SR) by linking domains, while maintaining a unified policy.
• BGPv4 (page 325) control-plane is supported on NSGs to enable learning/advertisement of LAN/WAN subnets
to/from the NSG.

2.2.3 Infrastructure Requirements

Key requirements for deploying Nuage VNS solution are described in this section. Refer to the latest Release Notes
for most up to date requirements.

2.2. VNS overview 12

 VNS User Guide, Release 5.3.2

Fig. 2.1: VNS Infrastructure Components

VNS deployment requires following infrastructure components in addition to the core elements, namely VSD, VSC,
and NSG:

Certificate Authority (CA)

EJBCA is a public Key Infrastructure consisting of a certificate authority (CA) that both issues and verifies the
digital certificates. The VNS process will rely on such capabilities in support of certificate based NSG authentica-
tion/authorization. Specifically, certificates are needed for Proxy and VSCs for communication related to notification
and bootstrapping. The VSD can use a self-signed certificate, but having a certificate from a certificate authority will
not require the client applications to process security warnings about unrecognised certificates.

Key Server (KS)

Key Server is a component for key management and is required for supporting the IPsec capability. In the current
release, the KS is assumed to be a trusted entity of the VSD. It is referred to as a managed KS whereby the enterprise
administrator manages the security policies for the organization, as well as the encryption policies on the KS. A KS
that is a separate entity from the VSD and is untrusted is referred to as an unmanaged KS and will be supported in a
future release.

Utility Host

A customer provided Utility Host that runs a Proxy, mail server and a notification application. The Proxy is utilized
for the definition, mediation and infrastructure level configuration functions for the NSG bootstrapping process. The
Proxy serves as a certificate-based proxy offering multiple https end-points. The host resides in the DMZ environment
and is the responsibility of the client.

VNS IP Network

The Nuage VNS can be used in any location with a IP network. The Nuage VNS actively participates in the IP routing
infrastructure. VSCs run OSPF or IS-IS for the IGP in addition to BGP. BGP is used to form a federation of VSCs and
to exchange information with external networks. In addition, BGP is also used to exchange routing information with
the data center provider edge router.

2.2. VNS overview 13

 VNS User Guide, Release 5.3.2

NTP Infrastructure

Because the Nuage VNS is a distributed system, it is important that the different elements have a reliable reference
clock to ensure the messages exchanged between the elements have meaningful timestamps. Nuage VNS relies on
each of the elements having clocks synchronized with Network Time Protocol (NTP).
The VSD relies on the NTP facilities provided by the host operating system. The VSC which is based on Alcatel-
Lucent SR OS has an NTP client. The NSG synchronizes to the VSC NTP.
Nuage recommends having at least two NTP reference clocks configured for each system with three reference clocks
preferred.

Domain Name System

In scaled Nuage VNS deployments, the VSD functional elements can be distributed across machines into clusters of
machines where the failover and load sharing mechanisms for the clusters rely on being referenced as a single DNS
entity. DNS naming of VSD and XMPP elements is also required for XMPP communication within the VNS. Since
the NSG needs to resolve Proxy Hostname, the DNS naming of Proxy is also required.

2.2.4 Virtualized Services Directory

The Virtualized Services Directory (VSD) is a programmable policy and analytics engine. It provides a flexible
network policy framework that enables network administrators to define and enforce the business policies being applied
across the network service in a user-friendly manner. The VSD enables:
• Centralized service policy definition and auditing for all endpoints
• Template-based service definition for intelligent endpoints
• Root and organization level permission-based multi-tenant systems
• Time-based automated endpoint configuration update
• Centralized software life cycle management
• Auto-discovery of intelligent endpoints
• Secure automated bootstrap of endpoints
• Centralized logging

Note: Since Nuage VNS extends the capabilities of the Nuage VSP solution, the Nuage Virtualized Services Platform
User Guide is referenced in this section for content that draws on the existing VSP/VSD capabilities.

VSD Role in VNS

The VSD plays a crucial role in facilitating auto-instantiation of service networking of NSG end-points in a VNS
deployment. The VSD drives the configuration modeling/mapping of NSG port/ VLAN resources to the Enterprise
defined services (Domain/Zone/Subnet/VPort). Service templates/instances are created independent of the NSG device
presence as described in the Nuage VSP User Guide. The infrastructure configuration necessary to get the NSGs
connected to the Nuage VNS in readiness for service configuration is also created in the VSD and is new to Nuage
VNS. Once the NSGs are activated at a site through a bootstrapping process, they automatically connect to the Nuage
VSD, download their service configuration and connect end-users to their services based on the pre-defined policies
with minimal end-user interaction. The NSGs serve as a policy enforcement end-point responsible for delivery of
networking services.

2.2. VNS overview 14

 VNS User Guide, Release 5.3.2

2.2.5 Virtualized Services Controller

The VSC acts as the control plane and main user (network administrator) interaction point for the NSG end-points. It is
a virtual machine application running all the control plane tasks. Based on Alcatel-Lucent’s robust and proven Service
Router Operating System, the VSC is able to provide a familiar management interface to the network administrator
and, has an industry-standard CLI as well as a familiar toolset such as SNMP management, syslog facilities and
TACACS+/RADIUS AAA facilities that allows the VSC to be easily integrated into existing data center monitoring,
alarming and logging infrastructure. The VSC is also responsible for running routing protocols like BGP that allow
it federate with other controllers as well as receive topology information of the underlay through Interior Routing
Protocols (IGPs)
The VSC is implemented as a virtual machine appliance. This means it can be hosted almost anywhere, only requiring
IP connectivity to the hypervisor and other Nuage VNS components, such as the VSD. The VSC provides the operator
with a console interface accessible both via an emulated serial port in the hypervisor as well as an SSH connection.
This virtual machine then is then configured to establish connectivity with the rest of the components of the Nuage
VNS solution.

VSC Protocols

OF-TLS

An OpenFlow (OF) session is established from the NSG to a specific VSC (or optionally, a pair of redundant VSCs).
The session is encrypted using Transport LayerSecurity (TLS), and this requires that both the NSG and VSC have
signed certificates for authentication.
A manual process (as explained in the Nuage VNS Install Guide) is used to generate certificates for each VSC via the
VSD. The certificates are signed by the Root CA. The signed certificates are then securely copied to the respective
VSCs, and each VSC is configured to enable Of-TLS. On the NSGs, the signed certificates are installed as part of VNS
installation. Post bootstrapping, the NSG instance initiates an OF session to a specific VSC and the signed certificates
are negotiated for authentication and for establishing a secure channel.
For a bootstrapped NSG, if OpenFlow connection to VSC is down for a configured period, then on regaining con-
nectivity NSG revokes its certificates, reboots and moves to a pre-bootstrapped stage. The configuration period for
auto-deactivation is set via VSD as part of provisioning.

XMPP

XMPP is the protocol used by the VSC to talk to the VSD to obtain the NSG policies and instantiation parameters.
The connection is established from the VSC and directly to the XMPP server component, using the management or
“out-of-band” connection of both these components. VSC will push through the XMPP bus the notification events
received by the VSC on the OF interface. An XMPP message is sent by the VSD in form of a group notification to
which the VSC nodes sub-scribe.

JSON-RPC

The light-weight protocol is used to communicate VSD-initiated configuration change notifications to the NSG on the
OF interface.

BGP/IGP

The VSC interacts with the underlay network, other VSCs in the data center and the PE routers at the edge of the DC by
using the BGP routing protocol. BGP is used to distribute the reachability information for each of the tenant services

2.2. VNS overview 15

 VNS User Guide, Release 5.3.2

and allows multiple VSCs to form a federation, allowing services to span outside of a single VSC. BGP is also used
to allow the direct connection of the VSC to the DC PE, eliminating any gateways or other equipment to do the PE
handoff. By participating directly in the BGP-VPNv4 address family, the VSC can exchange reachability information
between the different networking devices and allow the customer’s virtual network to be directly connected to wide
area VPN services offered by the DC operators. IGPs are also supported, both OSPF and IS-IS, where their main
purpose is to allow the VSC to receive routing and topology information of the underlay and properly establish the
BGP sessions to the DC PE router that normally uses its loopback interface for this purpose. It also allows the VSC to
track the status of the underlay network and reflect the tunnel status accordingly.

MP-BGP

TLS-encrypted MP-BGP sessions are used to communicate with the VSC when an NSG is configured with an access-
side BGP peer. There can be a maximum of two MP-BGP sessions for NSG-VSC communication — two on a single
uplink, or one on each uplink in a dual uplink scenario. The NSG uses the MP-BGP session to advertise BGP routes
learned from the downstream CPE to the VSC. The VSC then advertises these routes to remote NSGs. The NSG
programs the data path locally after learning BGP routes, and the VSC programs the data path for remote NSGs. BGP
IPv4 routes are translated to BGP EVPN routes before being advertised to the VSC.

2.2. VNS overview 16

 CHAPTER

THREE

OPERATIONS

• Getting Started with VNS (page 18)

• Bootstrapping (page 71)
• Gateway Operations (page 98)

17
 VNS User Guide, Release 5.3.2

3.1 Getting Started with VNS

• Overview (page 19)

• User Types (page 19)
– User Authentication (page 20)
• User Groups (page 20)
• Infrastructure Profiles in VNS (page 20)
• VNS Provisioning Overview (page 20)
• Detailed Provisioning Workflows (page 21)
– 1. Infrastructure Provisioning (CSPRoot Workflow) (page 21)

* Create Enterprise (page 21)

* Create Users (page 25)
* Global System Settings (page 25)
* Define Branding Attributes(Optional) (page 25)
* Define Underlays (page 26)
* Create Infrastructure Access Profile (page 27)
* Create Infrastructure Gateway Profile (page 29)
* Create Infrastructure VSC Profile (page 37)
* Define Rate Limiters (Optional) (page 38)
* Define Egress QoS Policies (Optional) (page 38)
* Define Address Translation Pools (Optional) (page 38)
* Create Network Services Gateway Template (page 38)
* Create Port Template (page 40)
* Create VLAN Template (page 42)
* Define Uplink Connections (page 43)
* Assign Infrastructure VSC Profile (page 43)
* Create/Define Auto-Bootstrapping Attributes (page 44)
– 2. Infrastructure Provisioning (Enterprise Admin Workflow) (page 50)

* Create Users within Enterprise (page 50)

* Instantiate Gateways for Enterprise (page 51)
* Configure TCP Maximum Segment Size (MSS) Adjustment (page 52)
* Add and/or Extend NSG Ports and VLANs (page 54)
* Define Uplink Connections (page 54)
* Define Branding for Gateway Activation Portal (page 60)
* Pre-Stage Gateway for Auto-Bootstrapping (page 60)

3.1. Getting Started with VNS 18

 VNS User Guide, Release 5.3.2

* Pre-Stage Gateway for One/Two Factor Bootstrapping (page 63)

– 3. Infrastructure Provisioning (Installer Workflow) (page 65)

* Bootstrap a Gateway (page 65)

– 4. Basic Service Configuration (Network Designer Workflow) (page 65)

* Typical Service Configuration (page 65)

– 5. Advanced Service Configuration (page 66)

* Network Uplinks (Wired and LTE) (page 66)

* NAT Traversal (page 66)
* Port Address Translation (page 67)
* PAT to Overlay (page 67)
* BiDirectional NAT (page 67)
* Application-Aware Routing (page 67)
* NSG Border Router (page 67)
* NSG Underlay Border Router (page 67)
* Egress QoS (page 68)
* IPsec VPNs (page 68)
* Access Resiliency (page 68)
* BGPv4 on NSG (page 68)
* Secondary IP on NSG (page 68)
* Service Chaining (page 69)
* DHCP Relay (page 69)

3.1.1 Overview

Nuage Networks Virtualized Network Services (VNS) enables delivery of business connectivity services to branch
locations regardless of size or geography. This chapter describes the infrastructure provisioning and service creation
workflows for deploying VNS and getting it started.
It is assumed that the Nuage VNS software components have been installed and configured for connectivity per the
procedure described in the latest version of the Nuage VNS Install Guide.

3.1.2 User Types

There are two user types:

• Administrator or Cloud Service Provider (CSP) users (hereinafter referred to as CSP Admin and CSPRoot) have
full visibility into all of the functionality of the VSD.
• Enterprise (or Organization) users belong to a specific tenant organization, and are organized into one or more
groups with different privilege and access levels:
– Root user or Enterprise Administrator is a consumer of the VNS services.

3.1. Getting Started with VNS 19

 VNS User Guide, Release 5.3.2

– Installer is a user responsible for receiving and commissioning an NSG at a branch location via bootstrap-
ping.
– Network Designer designs service templates that cover a set of specific service use cases. So whenever a
new service instance is needed, the associated service template can be instantiated. It is not uncommon for
the Enterprise Administrator to also be the Network Designer.

User Authentication

The VSD supports an LDAP interface for each enterprise for user authentication. Users for the enterprise are authen-
ticated against the given LDAP server. The User and Group hierarchy must be configured in VSD, with user names
matching the user name returned by LDAP.

3.1.3 User Groups

User groups are predefined user profiles that specify permissions given to users assigned to the group. There are user
groups created by default:
• BootstrapCA Group: Permits users to send pre-bootstrapping API requests to the VSD.
• CMS Group: Permits users to perform Cloud Management System operations.
• Everybody: Includes all users in the enterprise. This group permits users to perform user management tasks
only. This group is the default group for all newly created users.
• Operator Group: Permits users read-only access to all system data, including data within all organizations.
• Root Group: Permits users full read/write access to any function in the system.
• VSPCA Group: Permits users to perform NSG activation tasks and to send post-bootstrapping API requests to
the VSD.

3.1.4 Infrastructure Profiles in VNS

The concept of infrastructure profiles in VNS is used to reference a collection of infrastructure configuration parame-
ters that must be applied to a group of NSGs sharing a common set of attributes. For example, an operator may require
configurations to be created for:
• All NSGs connected via the Internet
• All NSGs deployed along US East Coast
• All NSGs connected via single DOCSIS uplink
• All NSGs that are part of a redundancy configuration
The infrastructure profiles are created at the CSP level and assigned to enterprise profiles. CSPRoot owns these
infrastructure profiles, and enterprise inherits the infrastructure profiles once they are instantiated. Enterprise are able
to override some infrastructure profile attributes on instance level only.

3.1.5 VNS Provisioning Overview

1. The CSPRoot creates the enterprise, users, and provisions infrastructure artifacts that an Enterprise Administra-
tor can consume for defining the enterprise network. Specifically the CSPRoot is responsible for creating:
• Enterprise(s)

3.1. Getting Started with VNS 20

 VNS User Guide, Release 5.3.2

• Enterprise users, user groups and managing permissions. An enterprise user with root privileges is referred to
as an Enterprise administrator
• Infrastructure gateway profiles, VSC profiles, Rate Limiters, etc.
• Templates for gateways and ports which are reusable constructs for consumption by Enterprise Administrators.
Gateways are instantiated from templates, and profiles are associated with templates.
• Infrastructure configuration, such as Key Server if an enterprise is enabled for encryption etc..
2. An Enterprise administrator is responsible for:
• Infrastructure provisioning for the enterprise, such as instantiating gateway(s) using the template(s) created by
the CSPRoot, facilitating gateway bootstrapping at branch locations, and provisioning connectivity, security and
resiliency constructs.
3. An Enterprise installer is responsible for:
• Receiving and bootstrapping a gateway at the branch location.
For auto-bootstrapping an installer does not need to be specifically identified or defined in the VSD.
4. An Enterprise network designer is responsible for:
• Configuration of services and advanced features such as L2/L3 VPNs, PAT/NAP services, service chaining,
security policies, WAN service connections, etc.. Networking abstractions domain, zone and subnet are used
to build the network service topology, and additional attributes further define how the VSD Policy Engine
will respond to service requests. All of the network building blocks are assembled within the context of an
“Enterprise”.

3.1.6 Detailed Provisioning Workflows

1. Infrastructure Provisioning (CSPRoot Workflow)

Create Enterprise

When the CSPRoot logs in, the screen shown in CSProot Login Screen (page 21) is presented. If the CSProot has
already created any Enterprises, they will be listed on the top left. In this view, there are no enterprises and the
CSProot must click on towards the bottom of the screen to create a new Enterprise using the popup shown in
New Organization Popup (page 22).

3.1. Getting Started with VNS 21

 VNS User Guide, Release 5.3.2

Fig. 3.1: CSProot Login Screen

3.1. Getting Started with VNS 22

 VNS User Guide, Release 5.3.2

Fig. 3.2: New Organization Popup

Enter an Enterprise Name, Description.

If the Admin Password is entered, then it serves as the password for the Enterprise Administrator for accessing the
VSD. Also, an admin is automatically created for the organization and no further action is needed to create the user.
Associate a previously created Organization Profile. Refer to the latest version of the Nuage VSP User Guide for
Configuring a New Organization Profile.

3.1. Getting Started with VNS 23

 VNS User Guide, Release 5.3.2

Fig. 3.3: New Organization Profile

Note: Specific to VNS, when an Organization Profile is created:

• Allow Gateway Management must be selected.
• If hierarchical shaping for Egress QoS is required, then it is necessary to select Allow Advanced QoS (to use
DSCP markings) and also specify the Available Forwarding Classes. Refer to the Egress QoS chapter for
details.
• Encryption Management Mode must be enabled for IPsec (page 403).
• Enable BGP checkbox must be checked for BGP (page 325) on NSG.
• Enable Performance Management checkbox must be checked for Application-Aware Routing (page 446) ca-
pabilities.

3.1. Getting Started with VNS 24

 VNS User Guide, Release 5.3.2

Create Users

When a new enterprise is created, the CSP administrator (CSPRoot) defines a user account and adds to the adminis-
trator group. Refer to Creating an Organization Admin User section in the Nuage VSP User Guide.

Note: This is optional if an Admin password is provided during Enterprise creation.

Administrators are able to create additional user accounts, and assign appropriate groups based on required access.
For more information refer to the section on Onboarding an Organization in the Using VSD Architect section of the
Nuage VSP Install Guide.
Users must also be created to generate the TLS certificates for the proxy and notification applications. The username
that is used is required to match – for example, if the certMgmt.sh script is executed with username of “proxy” then
a user named “proxy” must be created and added to the Root user group as part of the VSD configuration. This is
required as part of the certificate generation process for the VNS Proxy. Refer to the Nuage VNS Install Guide for
certificate generation details.

Global System Settings

There are VNS specific parameters that must be specified at the CSPRoot level in Settings -> System Configuration
. The common platform parameters are described in the Nuage VSP User Guide.
The VNS system configuration parameters are described in the context of the workflow where they are applied, specif-
ically:
• Branding (page 25)
• Two-Factor authentication (page 30)
• Auto-Bootstrapping (page 75)
• Group-Key IPsec (page 403) for defining global encryption attributes for configuration of the Key Server (KS).
- Map View Configuration (page 514)

Define Branding Attributes(Optional)

The CSPRoot can customize the organization logo that is used in the NSG’s activation portal during bootstrapping.

1. As CSPRoot, navigate to Settings -> .

2. In the System Configuration screen, scroll to the VNS section.

3.1. Getting Started with VNS 25

 VNS User Guide, Release 5.3.2

Fig. 3.4: Upload NSG Portal Logo

3. For the Default Activation Portal Logo, drag and drop a logo image in the Drop a Picture box. The image
will be used as the logo in NSG’s captive portal during bootstrapping, as well as in the email notification to the
installer. If not set, then the default Nuage Networks logo is used.
4. Select the Allow organizations to override portal logo checkbox to allow the organization’s avatar (if set) to
override the portal logo specfied in the previous step. Thus, the CSProot can allow an enterprise administrator to
select an avatar (page 60) to be used as an enterprise brand artifact in the NSG’s captive portal. If the checkbox
is selected but avatar is not set, then the default activation portal logo will apply.

Note: To allow the enterprise avatar to override the default portal, the Avatar Base URL in the System Configuration
screen must be updated from the default localhost address to the VSD address.

Fig. 3.5: Avatar Base URL

Define Underlays

Underlays are used to identify the different WAN networks that provide uplink connectivity to gateways in a VNS
deployment. The underlay objects are assigned when uplink connections are defined.

1. As CSPRoot, click on the Data Center Configuration icon .

2. Select the Infrastructure tab, and click on -> .

3. Click on towards the bottom of the screen to create a new underlay object which appears as a popup.

3.1. Getting Started with VNS 26

 VNS User Guide, Release 5.3.2

Fig. 3.6: New Underlay

4. Enter a Name and Description for the underlay and click Create.
Each object that is defined generates an internal underlay ID to distinguish the various underlays. By default Underlay
0 is always created which identifies any network. Underlays must only be defined if there is a need for disjoint underlay
connectivity.

Create Infrastructure Access Profile

Infrastructure Access Profile (IAP) contains the settings and authentication attributes that can be configured for con-
troling remote SSH connection to the NSG WAN port, as well as console access to the NSG. Specifically, SSH service
may be enabled/disabled on the WAN port, and password credentials may be customized for authenticating non-root
user nuage.

Note: SSH is enabled when an NSG is bootstrapped. In addition, Root login is disabled and user nuage is enabled
with a default password for SSH and console access. The IAP allows the behavior to be modified.

Refer to setting up secure access to NSG (page 108) for a more detailed description.
The Infrastructure Access Profile is assigned to a NSG Template (page 38) by the CSPRoot.

1. As CSPRoot, click on the Data Center Configuration icon .

2. Select the Infrastructure tab, and click on -> for Infrastructure Access Profile. The window displays
a list of existing profiles (if they exist). Click on towards the bottom of the screen to create a new profile
which appears as a popup.

3.1. Getting Started with VNS 27

 VNS User Guide, Release 5.3.2

Fig. 3.7: New Infrastructure Access Profile

Enter the following information:

• Name and Description for the profile.
• NSG system Username. Default is nuage.
• SSH Authentication Mode can be one of the following:
– Password Based

3.1. Getting Started with VNS 28

 VNS User Guide, Release 5.3.2

– Key Based
– Password and Key Based
• A new Password for the specified user. The password specified in the profile will be used for SSH login as well
as for console access.
• Enable or disable Source IP Filter. When enabled, remote access is allowed only from the specified IP ad-
dresses.
Click the Create button to generate the named profile.
Once the profile is created, select the SSH Key List and/or Source IP List tab to add the public keys and/or source IP
address list as required:
• SSH Key: Only RSA keys are supported.
• IP Address: Only IPv4 addresses are supported.

Create Infrastructure Gateway Profile

Infrastructure Gateway Profile (NSG Profile) contains gateway-level platform attributes inherited by gateways as they
are instantiated, connecting them to Nuage management infrastructure. Specifically the NSG Profile contains all the
attributes required for connecting a gateway NSG to the VSD for configuration synchronization, upgrades, security,
logging, monitoring etc.. The Infrastructure Gateway Profile is assigned to a NSG Template (page 38) by the CSPRoot.

1. As CSPRoot, click on the Data Center Configuration icon .

2. Select the Infrastructure tab, and click on -> for NSG Profiles. The window displays a list of existing
profiles (if they exist). Click on towards the bottom of the screen to create a new profile which appears as a
popup.

Fig. 3.8: New Infrastructure Gateway Profile

Enter the following information:

• Enter a Name and Description for the profile.

3.1. Getting Started with VNS 29

 VNS User Guide, Release 5.3.2

There are several tabs in the popup each of which must be completed.
Activation
• VSD Proxy FQDN identifies a DNS resolvable name of the proxy server that is used by the NSG during the
bootstrapping process. Refer to Proxy Reachability (page 102) for more details.
• Enable Two Factor Authentication when checked indicates that for bootstrapping, the installer will be authen-
ticated using two-factor authentication. This parameter applies to two-factor bootstrapping.

There are related Two Factor Authentication attributes in System Configuration settings ( -> Settings -> ) that
must also be specified by navigating to the VNS section. The attributes are:
• Two Factor Code Length specifies the code length (default 6 digits)
• Two Factor Code Seed Length is the length of the seed used for generating the random code
• Two Factor Code Validity is the duration in seconds for which the code is valid (default 300s)

Fig. 3.9: System Settings - Two Factor Authentication

Timers
Since the NSG can be deployed in an inherently untrusted environment, a number of security measures are imple-
mented to protect the network in the event an NSG is compromised or removed.
For scenarios where an NSG loses connectivity to all its controllers, the NSG’s operational state may be configured to
limit service impact while allowing for recovery from failure without compromising security.
Refer to NSG Security (page 103) for details since this section is specific to the configuration workflow.

3.1. Getting Started with VNS 30

 VNS User Guide, Release 5.3.2

Fig. 3.10: Infrastructure Gateway Profile - Timers

The attributes in the Timers tab may be configured to enable NSG operations modes as follows. All timers are
specified in ISO format (Days, Hours, Minutes):
• Stop all traffic forwarding for a certain duration and then revoke the NSG. Check Enable Auto Deactivation
and specify a duration (the default is 7 days). If not enabled, then it is equivalent to Never Deactivate (which
was an option offered in earlier releases) for trusted environments. The Auto Deactivation timer applies to two
NSG failure scenarios as described in NSG Security (page 103) section.

3.1. Getting Started with VNS 31

 VNS User Guide, Release 5.3.2

• Maintain full operational state for a certain duration. From the Controllerless Forwarding Mode dropdown,
select Local and Remote, and enter timer for Controllerless Remote Duration (the default is 3 days). Con-
trollerless Local Duration must also be configured when mode is set to Local and Remote.
• Maintain a limited operational state for a certain duration, whereby remote flows and tunnels are purged, while
retaining local flows with LAN-side forwarding. From the Controllerless Forwarding Mode dropdown, select
Local Only, and enter timer for Controllerless Local Duration (the default is 7 days). The Controllerless
Remote Duration will be greyed out when mode is set to Local Only.
• Withdraw all local and remote flows immediately. From the Controllerless Forwarding Mode dropdown,
select Disabled. If this option is selected, then local forwarding is maintained for 180s and remote flows are
purged.

Note: The various timers are additive. In other words, the timers must be specified such that the controllerless remote
duration is less than the controllerless local duration which must be less than the auto-deactivation duration.

Note: Only the default values of the timers have been qualified. However, you can configure values within the
following ranges:
• Auto Deactivation: 1 hour - 90 days
• Controllerless Local Duration: 2 hours - 90 days
• Controllerless Remote Duration: 1 hour - 3 days

• OpenFlow Datapath Synchronization Timeout (milliseconds) specifies the interval at which the kernel flows
are optimized using an algorithm for evicting flows. Default is 1000 ms. This is a read only OpenFlow tuning
parameter.
• OpenFlow Eviction Threshold (non-negative integer) is a limit on the number of flows that are cached in the
kernel. Once the threshold is reached, older flow entries are evicted. The eviction threshold presents a trade off
between the expense of maintaining large numbers of datapath flows, and the benefit of avoiding unnecessary
flow misses. Default is 2500 flows. This is a read only OpenFlow tuning parameter.
• OpenFlow Audit Timer (seconds) specifies the duration for which full operational state is maintained after
an NSG loses connectivity to all its controllers. After the audit timer expires, the NSG transitions into the
controllerless/auto-deactivation modes (depending on the mode configured). The audit timer be configured in
the range of 180 – 1800 seconds, but the default is 180 seconds.
Sync

3.1. Getting Started with VNS 32

 VNS User Guide, Release 5.3.2

Fig. 3.11: Infrastructure Gateway Profile - Sync

• Synchronize Configuration Days and Synchronization Time attributes create a timer that is maintained and
triggered at the NSG. It specifies how often the VSD is polled for configuration changes (including NSG soft-
ware upgrades). If two or more different configuration change sets are downloaded from the VSD, then the NSG
applies the newest set of configuration changes. Valid values are from once a week to once a day at a specified
time.

Note: The time entered has to take into consideration that it applies in the context of the NSG’s assigned time zone
(04:00 UTC and 04:00 PST are not the same).

The configuration reload and/or upgrade starts within 1 hour from the Synchronization Time unless Force Immediate
Synchronization is selected. This is to ensure that multiple NSGs are not making update requests to the VSD at exactly
the same second, and effectively spread the load.
Example: Synchronization Schedule can have days set at “Monday” and time set as 12:00 AM. Synchronization
window is set to 1 hour. Every Monday at a random time from 12:00 AM and before 1 AM local NSG time, the NSG
will attempt to get the configuration from the VSD and apply.

Note: The configuration synchronization schedule can be bypassed by a “Reload Configuration” option which pushes
and applies changes to NSG instantaneously. This option is available at the gateway instance level after the NSG is
bootstrapped. Select the NSG by navigating to Infrastructure -> NSG , and selecting Reload Configuration from
the rightmost panel.

3.1. Getting Started with VNS 33

 VNS User Guide, Release 5.3.2

Fig. 3.12: Apply configuration changes instantaneously

• Upgrade Policy is the policy for upgrading NSG software and is applied to the NSG as part of the configuration
synchronization. The policy options are:
– Apply update at window: It is the default option. When the policy is applied, if a new software version is
detected, then a download is initiated and the upgrade is performed at the next synchronization window.
Metadata URL is mandatory for this option.
– Update at bootstrap: This option is used when a mandatory upgrade is desired of a factory provided gold
image. It will initiate a download of the software, followed by an upgrade. Metadata URL is mandatory
for this option.
– Don’t update: No software download or upgrade is performed.
• Upgrade Metadata URL is the path to the upgrade metadata file. It is important to ensure that a fully qualified
path is entered (e.g. https://<server:port>/<path>/upgrade_metadata.json). Any arbitrary
port number may be used, as long as the same numbered port is opened on the proxy to redirect upgrade requests
from NSG to the web server.
Stats Collection

Fig. 3.13: Infrastructure Gateway Profile - Stats Collection

3.1. Getting Started with VNS 34

 VNS User Guide, Release 5.3.2

Currently, there is no separate Stats Collector Proxy. The proxy for NSG activation is also used for collecting stats.
The Stats Collector Port is the port on which the NSG will export the stats to the Proxy and defaults to 39090.

Note: Post bootstrapping, an NSG establishes a long lived TCP connection to the proxy for the purpose of stats
exporting.

In addition, stats collection also requires configuration on the VSD to specify the stats port to listen to. It is defined
at the CSPRoot level in the System Configuration ( -> Settings -> ) by navigating to the Statistics section of
attributes. There are two ports:
• Collector Port (with a default value of 29090) is being deprecated going forward. It is being retained only for
backward compatibilty.
• Collector Protobuf Port is the port opened by VSD for stats collection. The default is 39090.

Note: The Stats Collector Port in the infrastructure gateway profile must match the port in frontend statsconfig in
haproxy.cfg file, and the Collector Protobuf Port must match the port in backend vsdstat in haproxy.cfg. By default
both ports are set to 39090.

Fig. 3.14: System Settings - Stats Collection

NTP

3.1. Getting Started with VNS 35

 VNS User Guide, Release 5.3.2

Fig. 3.15: Infrastructure Gateway Profile - NTP

By default, the VSC is the NTP server (without any authentication) for the NSGs. To enable NTP server function on
the VSC, execute the following on the VSC under config>system>time>ntp

ntp-server
server 192.xxx.xx.xx
no shutdown

For security, an authentication key can be configured using the NTP Key Identifier and NTP Key fields in the
infrastructure profile. The NTP Key Identifier must be in decimal format within the range 0-255. The NTP Key must
be between 0 and 32 characters in length.
The following CLI commands may be used for enabling NTP and authentication to server:
1. Authenticate clients (i.e. NSGs) connecting to the server:
*A:vsc32>config>system>time>ntp# ntp-server authenticate
2. Ensure that the VSC NTP client does not authenticate towards the infrastructure NTP server (if the NTP server
is not using authentication):
*A:vsc32>config>system>time>ntp# no authentication-check
3. Apply the infrastructure profile key/password
*A:vsc32>config>system>time>ntp# authentication-key 1 key <NTP key> hash2 type
message-digest

Note: NTP synchronization across VNS components is mandatory to guarantee correct operation.

Logging

Fig. 3.16: Infrastructure Gateway Profile - Logging

3.1. Getting Started with VNS 36

 VNS User Guide, Release 5.3.2

• Logging State can be Remote Syslog or Disabled

• Server Address and Port of remote syslog server
Syslog is exported from the NSG using a secure TCP connection (TLS). Rsyslog is supported as a syslog server
destination. Server address and port may be exported via the proxy and as such the proxy address:port should be
specified in this field.
Once the information is completed, the CSProot can click the Create button to generate the named profile.

Create Infrastructure VSC Profile

The Infrastructure VSC Profile contains the IP addresses of the controller(s) in a VNS deployment. It is mandatory
for Network Port/VLAN to be associated with a VSC profile and a validation is done at the time of Bootstrapping an
NSG.

Note:
1. The VSC profile may be assigned by a CSPRoot to a VLAN template (of a network port template) or at the
VLAN instance level. It is recommended that network port VLAN template be assigned a VSC profile by
the CSPRoot.
2. The VSC profile is required for supporting the Dual Uplink feature. The controller topology and requirements
for assigning the VSC profiles to ports are discussed in more detail in the Dual Uplink (page 208) section.
3. Overlapping controller IP addresses are not supported on an NSG. Each configured VSC (single/dual uplink)
must have a unique IP address (even if in disjoint underlays).

1. As CSPRoot, click on Data Center Configuration icon and select the Infrastructure tab across the top.

2. In the far left side panel, click on the VSC Profiles icon . The Infrastructure VSC Profiles panel displays a
list of existing profiles (if they exist).

3. Click on for creating the first profile or on for adding a new profile. In either case, a popup for New
Infrastructure VSC Profile appears.
4. Enter the following in the popup.
• Name
• Description
• Probe Interval (milliseconds): Specifies the interval at which the VSC is probed for connectivity of OF TLS.
The default is 5000 ms, or 15000 ms for a scaled setup.
• Address Family: Select IPv4 or IPv6.
• First VSC: Defines the IP address of the first VSC that the NSG attempts to connect to using the network uplink
of the network port with which the profile is associated.
• Second VSC: Defines the IP address of the second VSC that the NSG attempts to connect to using the network
uplink of the network port with which the profile is associated.

Note: The following OpenFlow parameters on the VSC must match the configured probe interval: config
vswitch-controller open-flow gw-echo-time in seconds must equal the configured probe interval
config vswitch-controller open-flow gw-hold-time in seconds must equal 3 times the configured
probe interval

3.1. Getting Started with VNS 37

 VNS User Guide, Release 5.3.2

5. Click on Create to generate the named VSC profile.

Define Rate Limiters (Optional)

Rate Limiters are set of traffic management parameters describing a desired traffic profile. Rate-limiters are used by
QoS policies to enforce per Class of Service rate-conformance. Refer to Egress QoS (page 68).

Define Egress QoS Policies (Optional)

The Egress QoS Policy groups rate-limiting profiles, traffic directionality and classifiers to govern the rate of traffic
being sent or received by an end-host or application. The CSPRoot or Enterprise Admininstrator assigns the policy to
a port VLAN.
Refer to Egress QoS (page 68).

Define Address Translation Pools (Optional)

Address Translation Pools are a range of externally routable IP addresses. User or application traffic is translated prior
to being forwarded across the underlay network.
Refer to Port Address Translation (page 67).

Create Network Services Gateway Template

1. As CSProot, click on Data Center Configuration icon and select the Infrastructure tab across the top.

3. In the far left side panel, click on the Network Service Gateways icon . The Network Services Gateways
panel displays a list of existing templates (if they exist). Click on towards the bottom of the screen to create
a new NSG template which appears as a popup as shown in New NSG Template Popup (page 38).

3.1. Getting Started with VNS 38

 VNS User Guide, Release 5.3.2

Fig. 3.17: New NSG Template Popup

4. Enter a Name and Description.

5. Associate an Infrastructure Gateway Profile by clicking on the icon which will popup a window with a
list of previously created profiles. Select a profile for association.

Fig. 3.18: Select an Infrastructure Gateway Profile

6. Similarly, associate an Infrastructure Access Profile by clicking on the icon which will popup a window
with a list of previously created profiles. Select a profile for association.

3.1. Getting Started with VNS 39

 VNS User Guide, Release 5.3.2

7. Select a Gateway Personality as either:

• NSG as a branch appliance or,
• NSG-BR for the Border Router capability (page 178) or,
• NSG-UBR for the Underlay Border Router capability (page 165).

Note: LTE uplink connections are supported only for personality NSG.

8. Specify if SSH Service is to be Enabled or Disabled by selecting from the drop-down.

9. Specify if Instance SSH Override is Allowed or Disallowed by selecting from drop-down. The instance inherits
the template values and this flag controls whether an enterprise administrator can override the template settings
locally for a specific NSG instance.
10. Click on the Create button. The screen will refresh and the newly created template will be listed in the panel.

11. Once the template is created, another Infrastructure Gateway Profile may be associated by clicking on the
icon towards the right side of the screen which will popup a list of existing infrastructure profiles for association.

Note:
• A template cannot be deleted if there are any NSG, NSG-BR, NSG-UBR instances based on the template.
• Personality type may be changed in the template only if there are no instances or no BR connections.

Create Port Template

Refer to the Port allocation table (page 77) which lists the ports (on different form factors) that may be configured as
Network and Access ports.
1. Select an NSG template from the Network Services Gateway Panel.

2. At the bottom of the Ports panel, click on . The New Port Template popup appears.

3.1. Getting Started with VNS 40

 VNS User Guide, Release 5.3.2

Fig. 3.19: Create New Port Template

3. Name the port and add a Description if desired.

4. Enter a Physical Name.

Note: The Physical Name must match the naming convention of the ports on the NSG:
• For fixed Ethernet ports, the name must be port1, port2, port3 etc. (up to the number of ports supported by the
form factor)
• For removable interfaces (i.e. USB), the name must be lteX, where X is an integer (e.g. lte0, lte1) because
such an interface applies to an LTE uplink.

5. Select Type as Access or Network. For an NSG at most two ports may be assigned as network type ports. Even
if only one network uplink is being provisioned at the time of installation, it is recommended that another port
be reserved as a network type port for future use.

Note:
• An LTE uplink is only supported on a port of type Network.

6. Enter a VLAN Range for port. The default VLAN range is 0-4095.

Note: VLAN 4095 should not be used, and it is recommended that the default range be updated to a range between
0-4094 so that VLAN 4095 is not inadvertently selected.

3.1. Getting Started with VNS 41

 VNS User Guide, Release 5.3.2

Note: For LTE connection, only VLAN 0 (untagged) is supported.

7. Enter an MTU for the interface. Default is 1500 Bytes if not entered.
8. Select a Speed from the drop-down. Default is auto-negotiate if not entered.
9. Upon completion click the Update button. Once created the port templates will be listed in the NS Ports panel
of the NSG Template and each template can be individually edited ( ) or deleted ( ).

Attention: MTU and Speed specified at port template level will be inherited by the port instance. The values
inherited from the port template will become persistent if any changes are made, at instance level, to the inherited
values (MTU, Speed) or default attributes (e.g. NAT-T, Mnemonics), and will no longer receive template level
configuration changes.

Create VLAN Template

Note: VLAN template is optional and may be created if the CSPRoot wants to limit to specific VLAN tags.

1. Select a Port template from the NS Ports Panel.

2. At the bottom of the VLANs panel, click on . The New VLAN Template popup appears.

Fig. 3.20: Create VLAN Template

3. Add a VLAN Number and Description.

Note: For LTE connection, only VLAN 0 (untagged) is supported.

4. In the case of NSG-UBR, the VLAN template has an additional flag called Underlay Connection VLAN.
When enabled, it indicates that the VLAN will be associated with a underlay uplink and not a control uplink.
The latter requires a VSC profile to be associated with it.
5. Click on Create.
The policy will be inherited by the enterprise administrator who may be able to extend the VLANs (for access ports),
but cannot delete the VLANs in the template created by the CSProot.

3.1. Getting Started with VNS 42

 VNS User Guide, Release 5.3.2

Define Uplink Connections

To define an Uplink Connection:

1. Navigate to the VLAN Template at Gateway template level for a port template of type network.

2. Click on the connection icon from the upper right action icons.
3. The workflow for configuring the uplink connection attributes is exactly as described in Define Uplink Connec-
tion (page 55) at the instance level.

Fig. 3.21: Define Uplink Connection at VLAN Template

Note: Uplink Connections defined at VLAN template level are inherited at VLAN instance level. However, user can
set a new uplink connection at instance level.

Assign Infrastructure VSC Profile

Note:
• Associating a VSC profile to VLAN template is optional. However, VSC profile must be assigned to at least
one network port VLAN when an instantiated NSG is to be Bootstrapped. Only a CSProot can assign a VSC
profile at the instance level.
• Thus, it is recommended that VLAN template be created by CSProot for all port templates of type Network
and the VSC profile be associated at the template level itself to streamline the workflow for the Enterprise
Administrator.

3.1. Getting Started with VNS 43

 VNS User Guide, Release 5.3.2

To assign a VSC profile:

1. Navigate to the VLAN Template at Gateway template level for a port template of type network.

2. Click on the VSC profile icon from the upper right action icons.
3. The panel below the icon will either display a VSC Profile already assigned or will allow you to assign a VSC
Profile. Select the paper clip icon which will popup a list of defined Infrastructure VSC Profiles. Choose a
profile and click on the Select button in the popup to complete the association of the VSC profile to the VLAN
template.

Fig. 3.22: Assign Infrastructure VSC Profile to VLAN Template

Attention: VSC profile and Egress QoS profile defined at VLAN template level are inherited at VLAN instance
level. However, any changes to VLAN instance attributes, such as PAT pools association, cause the inherited VSC
profile and Egress QoS profile links to persist on the instance. Thus, subsequent changes at template level will not
be propagated to the modified instance.

Create/Define Auto-Bootstrapping Attributes

Note: Auto-Bootstrapping only applies to gateways of personality type NSG.

Refer to Auto-Bootstrapping (page 73) for an overview of the process. The CSPRoot functions related to auto-
bootstrapping, involve:
• Create Auto Bootstrap Data that enables an NSG to initiate auto-bootstrap request. The Auto Bootstrap Data
created by CSPRoot does not have an enterprise context.
• Review and assign/reject pending bootstrap requests
• Define match criteria for auto-assignment of requesting NSGs
The workflow is as follows:

3.1. Getting Started with VNS 44

 VNS User Guide, Release 5.3.2

1. As CSPRoot, select the Data Center Configuration icon , and navigate to the Infrastructure tab -> Network
Service Gateway -> Auto Bootstrap for Auto Bootstrap functions.

Fig. 3.23: CSPRoot Auto Bootstrap Functions

Create Data for Auto-Bootstrapping

The CSPRoot may create the Auto Bootstrap Data file which is required by the NSG for initiating the auto-
bootstrapping request. The file is created and downloaded and consumed by the form factors as follows:
• Downloaded to a USB for physical NSG device
• Added as a CD-ROM device for NSG-V on KVM. The CD-ROM option must be specified in the XML (refer to
NSG-V on KVM installation instructionsin the VNS Install Guide)
• Added while manually launching an Instance in AWS Dashboard or via the CloudFormation JSON file for
NSG-AMI

1. From the panel, select Download Auto Bootstrap Data which will open the dialog for creating the Auto
Bootstrap Data file.

3.1. Getting Started with VNS 45

 VNS User Guide, Release 5.3.2

Fig. 3.24: Create Auto Bootstrap Data

2. Select the form factor for Allowed Network Services Gateway Type from one of:
• All
• NSG-AMI
• NSG-E
• NSG-V
3. Download Type is either ISO (if NSG Type is All, NSG-E, NSG-V) or YAML (if NSG Type if NSG-AMI).
4. Select the Network Services Gateway Template. When the Auto Bootstrap Data file is created, some informa-
tion is selected from the template specified.

3.1. Getting Started with VNS 46

 VNS User Guide, Release 5.3.2

5. Select the Download button to create the Auto Bootstrap Data file.
When the CSPRoot creates the Auto Bootstrap Data, there is no enterprise context, and hence a requesting NSG will
be placed in pending state (in unassigned state) for review by CSPRoot so that it can be assigned to an enterprise. The
CSPRoot may optionally define the match criteria to auto assign (page 48) a requesting NSG to an enterprise.

Review Pending Bootstrap Requests

When an NSG initiates a request for auto-bootstrap and the Auto Bootstrap Data file has no enterprise context, then
the request is registered in the VSD as a pending request, in unassigned stated, for review by the CSPRoot.

1. From the auto-bootstrap panel, select Pending Requests which will display a list of pending requests and
the corresponding state. Select one in unassigned state, which will popup a screen with the requesting NSG’s
fingerprint and allow the CSPRoot to assign the requesting NSG to an enterprise or reject the request.

Fig. 3.25: CSPRoot Review of Pending Bootstrap Request

2. To assign, click on icon towards the bottom of the popup, and select an enterprise. Click on Assign to
update. The status of the NSG in the pending request list will change to assigned. In addition. the NSG will be
added to the pending list at enterprise level for admin review.
3. The CSProot may also reject the request in which case the bootstrapping process on the NSG will exit. The
status in the pending request list will change to denied until clean-up (page 75).

3.1. Getting Started with VNS 47

 VNS User Guide, Release 5.3.2

Define Enterprise Auto Assignment Criteria

This function allows the CSPRoot to define match criteria to auto-assign requesting NSGs to an enterprise. NSGs that
get matched will be reflected in the CSPRoot Pending Requests list with an assigned status, and will also be placed in
the enterprise admininstrator’s Pending Requests list for review.

1. Select Enterprise Auto Assignment which will list existing match rules (if any) or allow you to edit existing
rules or add a new rule. The popup for specifying the match attributes is shown in the figure below.

3.1. Getting Started with VNS 48

 VNS User Guide, Release 5.3.2

Fig. 3.26: Enterprise Auto Assignment

2. Enter a Name and Description for the match definition.

3. Enter a Priority for the match rule to specify an order in which multiple attributes must be matched (if specified).
A lower value implies a higher preference. If a value is not specified, then the system will assign a value.

3.1. Getting Started with VNS 49

 VNS User Guide, Release 5.3.2

4. Select an attribute and enter one or more values to match on. Attribute options are:
• Hostname
• Serial Number
• IP address
• MAC address

5. Click on icon to select an enterprise. If the match is successful, then the requesting NSG will be auto-
assigned to the selected enterprise.

2. Infrastructure Provisioning (Enterprise Admin Workflow)

An enterprise administrator via the VSD has visibility into all the sites in the enterprise. The enterprise administrator
instantiates gateways for the enterprise and facilitates the site onboarding process when NSGs have to be activated at
branch loctions.

Create Users within Enterprise

Enterprise administrator is required to create installer users so that they can bootstrap the NSGs at the branch locations.
1. Enterprise admininstrator is logged in as a VSD user.

2. Select the Settings tab across the top of the screen. In the far left panel, click on Users icon .

3. Click on towards the bottom of the screen for adding a user. A popup appears as shown in Create an Enter-
prise User (page 51).
3. Enter all the details in the popup.

Note: Email is used as part of initial notification to installer and serves as the first factor for authentication of NSG
install. Mobile is used as part of a 2nd factor challenge soliciting a user to confirm his/her identity. Mobile number
should be entered as <+><country code><mobile number>. Password is used by user for logging into VSD and is
optional. Typically an installer will not log into the VSD.

3.1. Getting Started with VNS 50

 VNS User Guide, Release 5.3.2

Fig. 3.27: Create an Enterprise User

Instantiate Gateways for Enterprise

All CSPRoot created NSG gateway templates are available to Enterprise to instantiate from. NSG instances will
inherit all the configuration of the template they are associated with. Furthermore, as NSG gateway templates are
edited/updated, changes are applied across all associated instances.
1. Enterprise administrator is logged in as a VSD user.
2. Select the Infrastructure tab across the top of the screen. In the far left side panel, click on the Network Service
Gateways icon .

3. Click on at the bottom of the screen which displays a popup for creating an NSG instance:
Create an NSG Instance
4. Enter a Name and Description.
5. Select a Form Factor Template from the drop-down of all available templates.
6. Configure SSH Service. By default it is set to Inherited and may be changed to Enable or Disable only if the
CSPRoot has allowed override. An error appears if a change is attempted when the CSPRoot has not allowed
override.
7. Configure a Control Traffic COS and Control Traffic DSCP to assign remarking values for self-generated
control traffic.
8. The Device Shipping Address is optional and specifies the installation location.
9. The Time zone specifies the time zone of the NSG instance. The supported formats are UTC + or - offset,
Country/Region, and Continent/City. The exact values supported are as found for Linux system installation
(/usr/share/timeinfo). DST is implicitly handled when using “Continent/City” format. The value entered in this
field impacts the Synchronization Schedule value (see System Synchronization and Upgrade (page 32)).

10. To assign an installer, click on icon towards the bottom of the popup. Another popup appears as shown in
Select an Installer (page 52). An installer can be elected from the available list.

3.1. Getting Started with VNS 51

 VNS User Guide, Release 5.3.2

Fig. 3.28: Select an Installer

Note: An installer does not necessarily have to be assigned at the time of gateway instantiation if the gateway is not
ready to be shipped. To assign an installer post gateway instantiation, refer here (page 63).

11. If “Notify installer right away” is checked, then a notification event is generated whereby an application (e.g.
Mail server) can be used to generate email to the installer at that point. This option should be used if the gateway
is ready to be shipped immediately. Typically there may be a lag between the time the gateway is instantiated
and shipped.

Configure TCP Maximum Segment Size (MSS) Adjustment

On an NSG instance, an Enterprise administrator can, optionally, configure a TCP Maximum Segment Size (MSS)
adjustment. It specifies the maximum amount of data that a host (receiver) is willing to accept in a single TCP/IP
datagram from the sender. When TCP session are established between hosts, the 3-way handshake establishes the
MSS for the session. Each side of a TCP connection reports its MSS value to the other side. This is typically done
to avoid fragmentation of the TCP packets within the transient network. The feature does not improve fragmentation
issues for non-TCP traffic.
When configured, the network nodes modify the MSS value in the TCP SYN and SYN-ACK packets to the lower of
the requested and configured MSS values.

3.1. Getting Started with VNS 52

 VNS User Guide, Release 5.3.2

In the case of NSGs, by carefully selecting the MSS adjustment value, TCP packet fragmentation can be avoided at
the NSGs, as well as in the underlay network.
Example below describes how the value may be selected:
• For both WAN/uplink overlay traffic of VxLAN and IPsec, assume ~130 bytes of overhead (depends on crypto
options)
• 40 bytes for TCP header
• MTU 1500
In such a scenario, the MSS adjustment value may be set to 1330 bytes (1500-130-40).

Fig. 3.29: Configure TCP Maximum Segment Size

The TCP MSS is configured as follows:

1. Navigate to an NSG instance and select edit option.

3.1. Getting Started with VNS 53

 VNS User Guide, Release 5.3.2

2. In the popup, select Enable Maximum Segment Size checkbox, and enter the TCP Maximum Segment Size
value. Range is 512-9000 bytes.
3. Click Update

Note:
• TCP MSS adjust value must be less than the MTU defined on the uplink ports. The VSD does not enforce the
validation.
• Configuration and/or modification of TCP MSS Adjust is service impacting.
• The value applies to all uplinks, all uplink port types, and all gateway personality types (NSG/NSG-BR/NSG-
UBR). In other words, the configuration is not specific to an uplink MTU or a particular underlay.
• Currently there is no support for Path MTU discovery capability.

Add and/or Extend NSG Ports and VLANs

Once the gateway(s) is instantiated, the ports/VLANs are inherited from the NSG Template. If the VLAN exists at the
template, then it will be automatically created at the instance. Otherwise, it will have to be manually created once a
new Network Port is added to the NSG Instance.

Attention: Post-bootstrapping any changes to network VLAN configuration will be service impacting, especially
interworking with IKE and BGP where uplink information is shared with an upstream peer. In addition, statistics
associated with the VLAN will be lost. If there is a business reason for changing VLAN configuration, then ensure
that both old and new VLANs are available in the underlay until service is restored on the uplink, after which the
old VLAN may be removed.

The enterprise administrator may also add ports and/or extend VLANs (for access ports).

Note:
• NAT Traversal is a port level attribute that must be specified by the enterprise administrator. Refer to NAT-T
provisioning (page 143) for detailed workflow.
• For a network port with an LTE uplink, NAT Traversal (page 206) may need to be enabled if the provider of
the LTE device has assigned a private IP address.

Define Uplink Connections

When an NSG network port is instantiated, the uplink connection is inherited from the template (if defined) or will
have to be manually created with attributes that matches the deployment environment.
Configure the following parameters:
• Connection specifies the type of uplink connection. The values can be Dynamic (default), Static,
PPPoE, or LTE.

Note: Refer to LTE uplink (page 202) feature overview for LTE connection mode.

3.1. Getting Started with VNS 54

 VNS User Guide, Release 5.3.2

• Role can be Primary or Secondary. The default is Primary for first uplink, and Secondary for the
second uplink.

Note: Refer to Uplink Role (page 207) section for how the tagging is used for network link redundancy.

• Address Family: Select IPv4 or IPv6.

Note: Only static, dynamic, and LTE uplink connections are supported for IPv6.

Note: If a VSC infrastructure profile with IPv6 is assigned to an uplink VLAN, only IPv6 can be selected
for the Address Family parameter.

Note: IPv6 must be enabled at the domain level before IPv6 can be selected for an uplink connection.

Note: NAT probes are enabled by default on network ports, but NAT traversal is not supported for IPv6.
The probes must be disabled before configuring an IPv6 uplink connection. Disable the NAT Probes
Enabled parameter on the NSG port configuration form before creating an IPv6 uplink connection. Refer
to Configuring NAT-T settings at the NSG Port level (page 143) for more information.

• Underlay specifies the WAN network over which the connection is being defined. If not specified,
then Underlay 0 is assumed. It is required to be specified for NSG-UBR (page 165) feature for
underlay uplinks.
• Reachability Verification specifies the method for verifying reachability on the specified connec-
tion (control uplink or underlay uplink).
• Download rate limit (Mbps) specifies the maximum rate at which data can be downloaded through
the uplink.
• Underlay NAT enables underlay NAT on the uplink. If underlay address translation is enabled at
the domain, zone, or subnet level, traffic is sent from the uplink with NAT.
• Underlay Routing enables underlay routing on the uplink. If underlay routing is enabled at the
domain, zone, or subnet level, traffic is routed from the uplink to the network underlay.

Note: Refer to Per Uplink NAT (page 566) for more information about enabling underlay NAT or
underlay routing at the uplink level.

Note: A reload config is required after configuring the Underlay Routing and Underlay NAT parame-
ters.

Uplink Configuration Workflow

1. As enterprise administrator, navigate to Infrastructure -> NSG instance -> Network Port -> VLAN.

2. Click on the connection icon from the upper right action icons.

3.1. Getting Started with VNS 55

 VNS User Guide, Release 5.3.2

3. Select Connection mode as either Dynamic, Static, PPPoE, or LTE. Depending on the selection, additional
parameters must be configured as shown below:
Static Mode Parameters
Enter the IP addresses for the Network, Gateway, and DNS server. DNS server is required for static
uplink connection type.

Attention:
• For NSG-UBR, for underlay uplinks, Connection mode is Static, but DNS server is not required.
• Ensure that the IP address entered is fully resolvable. An invalid IP address will result in traffic
loss on the uplink.
• Post-bootstrapping, if the IP address is changed via a reload configuration, then the correspond-
ing peer configuration for BGP/IKE (as applicable) will also need to be updated.

Fig. 3.30: Static Connection Mode Configuration

PPPoE Mode Parameters

Configure the Installer-Managed parameter, if required. If the parameter is selected, PPPoE credentials
are specified by the installer during NSG bootstrapping.
If the Installer Managed parameter is not selected, enter the ISP-provided Username and Password.
Consider the following for PPPoE credentials:
• If the PPPoE connection is not installer-managed, the credentials can be configured post-
bootstrapping. A reload config is required for these changes to go into effect.
• The Installer Managed parameter cannot be modified after the NSG is bootstrapped. If the PPPoE
connection is installer-managed, the PPPoE credentials also cannot be modified once the NSG is
bootstrapped. The PPPoE credentials and management mode can be changed only by revoking the
NSG and bootstrapping it again.
• If the Installer Managed parameter is selected, auto-bootstrapping is not supported.
• If the Installer Managed parameter is selected and the NSG is configured with a dual-uplink con-
nection, bootstrapping may succeed through the non-PPPoE uplink even if the credentials on the
PPPoE uplink are incorrect. To avoid this scenario, the installer must bootstrap the NSG through
the PPPoE-configured port only. When PPPoE configuration is successful, the installer should then
plug in the non-PPPoE port and proceed with bootstrapping.
See Installer Instructions (page 77) for information on the NSG bootstrapping workflow.

3.1. Getting Started with VNS 56

 VNS User Guide, Release 5.3.2

Fig. 3.31: PPPoE Connection Mode Configuration

LTE Mode Parameters

• Select an Interface option. Select Automatic for USB dongle-based and integrated LTE uplinks.
• Enable the Auxiliary Uplink checkbox to configure the LTE uplink as a circuit of last resort. When enabled, the
LTE interface is activated only when the wired connection goes down. For more details, refer here (page 207).

Fig. 3.32: LTE Connection Mode Configuration

• Once an LTE uplink is created, additional attributes (e.g. SIM Card APN and PIN) may be specified
via a Custom Property object (for modem-based or integrated devices) that appears at the bottom
of the connections panel.
The information is in the form of a Name:Value pair. Once the information is pushed to the NSG, it will
replace the default values present in the LTE Configuration file used for dialing modem based devices. It
is recommended that custom properties be entered at the instance level.
The following example shows custom property being added (as a name:value pair) for configuring APN
settings (default APN is Internet).

3.1. Getting Started with VNS 57

 VNS User Guide, Release 5.3.2

Fig. 3.33: LTE Custom Property

The following example displays another custom property that is added for configuring PAP authentication
instead of CHAP (which is the default).

3.1. Getting Started with VNS 58

 VNS User Guide, Release 5.3.2

Fig. 3.34: LTE Custom Property Example

Note:
• PPPoE can only be selected for one uplink.
• LTE can only be selected for one uplink.
• The “apn” attribute-value pair is supported for integrated LTE uplinks. The system automatically determines
the APN to dial in most cases and it is not required to specify the APN using a custom property. In some cases,
if the LTE connection fails for your carrier or region, the automatic APN settings may not be the correct one.
Please contact your LTE SIM provider for the APN required to dial for that connection. You can then configure
that value as a custom parameter. For example, name: “apn” and value: <carrier provided apn>. In this case,
the configured APN will override the system default APN.
• “pdp-type” is a configurable custom parameter that determines the PDP type associated with the LTE connection.
The PDP type is automatically determined by the NSG and specifying it as a custom property is not required
or recommended. However, you can specify “pdp-type” as a custom property and override the automatically
determined value.
• If the mode is set to Dynamic, Static or PPPoE, and WAN check fails, then bootstrapping is not allowed to
proceed until another registration link with the correct uplink connection configuration is received.
• The mode selection and static/PPPoE/LTE configuration (if specified) is included in the registration link that is
sent to the installer, and bootstrapping uses the information for WAN checks.
• Post-bootrapping, mode cannot be changed.
• If the mode is set to Dynamic, 31-bit prefixes are supported. However, the gateway address must be specified
for DHCP connections to function properly.

3.1. Getting Started with VNS 59

 VNS User Guide, Release 5.3.2

4. Select Role as Primary or Secondary.

Attention: For NSG-UBR, for underlay uplinks, Role is None.

5. Select an Underlay from the list.

6. Select Reachability Verification to select the method for verifying reachability on the specified underlay. The
options are Operational Link, Control Session, and BFD.

Note: The following are the valid selection options:

• For all NSG form factors, for control uplink connections, Control Session must be selected. The reachability
will be based on the status of the OpenFlow connection.
• For NSG-BR, for BR-connections, Operational Link or BFD must be selected. Refer to Underlay Reachability
(page 183).
• For NSG-UBR, for underlay uplinks, Operational Link or BFD must be selected. Refer to BFD configuration
workflow (page 171).

Define Branding for Gateway Activation Portal

If the CSPRoot has allowed the organization to override the default activation portal, then the enterprise administrator
must select an avatar by dragging the desired logo into the organization object as shown in the figure below with the
example Nokia logo.

Fig. 3.35: Select an Avatar for Organization

Pre-Stage Gateway for Auto-Bootstrapping

Refer to Auto-Bootstrapping (page 73) for an overview of the process. The admin functions related to auto-
bootstrapping, involve:
• Create Auto Bootstrap Data that enables an NSG to initiate auto-bootstrap request. The Auto Bootstrap Data
created by an admin has an enterprise context.

3.1. Getting Started with VNS 60

 VNS User Guide, Release 5.3.2

• Review and approve/reject pending bootstrap requests

• Define match criteria for auto-approval of requesting NSGs
The workflow is as follows:

1. As enterprise administrator, navigate to the Infrastructure tab -> -> for Auto Bootstrap functions.

Fig. 3.36: Enterprise Admin Auto Bootstrap Functions

Create Data for Auto-Bootstrapping

The enterprise admin may create the Auto Bootstrap Data file which is presented to the NSG on a USB (for physical
NSG device) or added as a CD-ROM device (for NSG-V), and is used by the NSG for initiating the auto-bootstrapping
request.
The process is identical to the steps described in the CSPRoot workflow (page 45), except that the Auto Bootstrap
Data file will be created within an enterprise context. The requesting NSG will be placed in pending state (in assigned
state) for review by enterprise admin so that it can be approved and assigned to an instance. The admin may define the
match criteria to auto approve (page 62) the requesting NSG to an instance.

Review Pending Bootstrap Requests

When an NSG initiates a request for auto-bootstrap and the Auto Bootstrap Data file has an enterprise context, then
the request is registered in the VSD as a pending request (in assigned state) for review by the enterprise admin.
1. Select Pending Requests which will display a list of pending requests and the corresponding state. Select one
in assigned state, which will popup a screen with the requesting NSG’s fingerprint and allow the admin to assign
the requesting NSG to an instance or reject the request.

3.1. Getting Started with VNS 61

 VNS User Guide, Release 5.3.2

Fig. 3.37: Admin Review of Pending Bootstrap Request

2. To assign, click on icon towards the bottom of the popup, and select an NSG instance. Click on Approve to
update. The status in the pending request list will change to approved until clean-up (page 75).
3. The admin may also reject the request in which case the bootstrapping process on the NSG will exit. The status
in the pending request list will change to denied until clean-up (page 75).

Define Instance Auto Approve Criteria

The enterprise administrator may define match criteria to auto-approve requesting NSGs to an instance. When a
requesting NSG is matched to an instance, it is deemed as approved and bootstrapping will begin on the requesting
NSG. NSGs that get matched will be reflected in the Pending Requests list with an approved status.

1. Select an NSG instance and in the right panel select for bootstrap and find the Auto Bootstrap section to
define the match rule for auto approve.

3.1. Getting Started with VNS 62

 VNS User Guide, Release 5.3.2

Fig. 3.38: Instance Auto Approve

2. Select a Match Attribute and enter a Match Value.

Pre-Stage Gateway for One/Two Factor Bootstrapping

The following describes the requirements for normal one-factor or two-factor bootstrapping process.
When a new gateway is to be shipped to a site for onboarding, the enterprise admininstrator has to:
• Assign an installer (page 63) who can receive and activate the NSG at the targeted site, and
• Notify the installer (page 64) to initiate bootstrapping

Assign Installer to Gateway

If an Installer is not selected at the time the Gateway is instantiated or a different installer is required, then the assign-
ment can be done when the gateway is ready to be shipped.
1. Select a Network Service Gateway instance

2. Select the Bootstrap configuration option on the right .

Fig. 3.39: Assign an Installer

3.1. Getting Started with VNS 63

 VNS User Guide, Release 5.3.2

3. If an installer has been assigned, the name will be displayed, otherwise “No Installer” will be displayed.

4. Click on icon which will popup a list of installer names from which a name can be selected.

Notify Installer - Send Gateway Registration Info

When the gateway is ready to be shipped and deployed, the selected installer must be notified of the registration
information. The registration information includes an identifier for the gateway appliance to ensure that the intended
gateway is activated from the site. The installer is notified using the following steps:
1. Select a Network Service Gateway instance

2. Select the Bootstrap configuration option on the right .

3. Assuming that an installer has been identified, the following will be displayed.

Fig. 3.40: Notifying an Installer

4. Clicking on “Send Activation” button will generate a notification event whereby an application (e.g. Mail server)
can be used to generate an email to the installer.

Note: If a VSC profile is not associated with the uplink port/VLAN of the instantiated NSG, then Send Activation
will result in a server error indicating VSC Profile not assigned to network port. The enterprise administrator will need
to inform the CSProot of the error.

The installer receives an email with a link for registration. After the installer clicks on the registration link, if two-
factor authentication is enabled in the Infrastructure profile, then a confirmation code is also generated. The code can
be consumed using a mechanism, such as text to SMS gateway (to send to installer’s mobile device), to authenticate
the installer.

3.1. Getting Started with VNS 64

 VNS User Guide, Release 5.3.2

3. Infrastructure Provisioning (Installer Workflow)

An installer at a site is responsible for receiving and bootstrapping the Gateway.

Bootstrap a Gateway

Bootstrapping is a key installation capability to securely commission NSGs into the VNS network.
Refer to Bootstrapping (page 71) for the detailed Installer workflow for on-site onboarding of the NSG using one-
factor or two-factor bootstrapping process.
Refer to Auto-Bootstrapping (page 73) for on-site staging of NSG for self-installation with minimal installer interven-
tion.

4. Basic Service Configuration (Network Designer Workflow)

Nuage VNS features a number of networking abstractions that act as building blocks to define the connectivity within
a network overlay service in an Enterprise. Specifically:
• Domains are the abstraction for either a switching (L2) or routable (L3) network service. Multiple domains
can exist per Enterprise. - Within a L2 domain, Nuage VSP establishes connections based on MAC addresses
contained in every frame.
– Under a L2 domain, vPorts are defined. vPort is the logical representation of an interface or attachment
point. In VNS, two types of vPorts can be defined: bridge, host.
Unmanaged L2 domains are supported.
– Within a L3 domain, Nuage VSP manages the IP networking layer and forms a single “L3 space”. In
standard Networking terminology this corresponds to a VRF instance. Since Nuage VSP manages the L3
space, it establishes direct connections between guest VMs based on MAC and/or IP information, also
known as distributed routing.

* Under a L3 domain, zones are defined. A zone does not map to anything on the network directly, but
instead it acts as an object with which policies are associated such that all endpoints in a zone share
the same set of policies.

* Under a zone, subnets are defined. A subnet is a specific Layer 2 subnet within the Domain instance.
A subnet is unique and distinct within a L3 domain. That is, subnets within a Domain are not allowed
to overlap or to contain other subnets in accordance with the standard IP subnet definitions. In VNS,
a subnet is fully owned by an NSG.

* Under a subnet, vPorts provide a more granular configuration and also support a split workflow, where
a vPort is configured and associated with a gateway port before the port exists on the gateway. This is
the lowest level where properties such as ACLs, QoS can be defined. Host and Bridge Interfaces are
attached to vPorts. When a vPort created and attached to a subnet, there will be a validation check to
ensure that the vPort being attached belongs to a gateway that owns the subnet. If it does not, then the
attachment is not allowed.

Typical Service Configuration

Routing and Switching

The following illustrates a minimal workflow for bringing up VNS service assuming there are two sites with one
bootstrapped NSG at each site. Advanced configuration can be applied, once the basic configuration is completed:

3.1. Getting Started with VNS 65

 VNS User Guide, Release 5.3.2

1. Create L2 or L3 domain template. For a minimal configuration:

• Add an ingress security policy to allow all traffic (refer to VSP User Guide) since by default it is set to
deny all.
2. Instantiate domain
3. For L2 domains:
• Attach two vPorts.
• If vPort type is bridge, then specify DHCP address pool for IP address allocation/management (refer to DHCP
on Bridge Port (page 237)).
4. For L3 domain:
• DHCP Relay configuration is available at L3 Domains. Refer to DHCP Relay (page 69) for feature workflow
details.
• Create zones (at least one for a minimum VNS configuration)
• Create subnets (at least two subnets, one per NSG, for a minimum VNS configuration)
• Attach vPorts (minimally one vPort to each subnet)
– If vPort type is bridge, then specify DHCP address pool for IP address allocation/management (refer to
DHCP on Bridge Port (page 237)).
By default the Ingress ACL policy is set to Deny all traffic. At a minimum to test the service configuration, the policy
must be updated to allow all traffic. Refer to Ingress Security Policies in the VSP User Guide.

5. Advanced Service Configuration

Network Uplinks (Wired and LTE)

An NSG may be provisioned with up to two network uplinks for data/control path redundancy as well as to allow traffic
to be routed over two transport networks (e.g. IPVPN and Internet). Ehternet-based and LTE uplinks are supported.
Provisioning includes:
• Defining uplink role as Primary or Secondary
• Configuring VSC Profile depending on controller topology and redundancy considerations.
• Configuring forwarding policy for indicating Uplink Preference
Refer to Network Uplinks (page 201) chapter.

NAT Traversal

The capability enables Internet connected NSGs that are front-ended by NAT devices to be part of the VNS network.
The NAT-T feature allows VNS data and control plane connections to traverse the NAT devices.
Provisioning entails a network port level attribute that is configured as part of the infrastructure provisioning by an
enterprise administrator.
Refer to NAT Traversal (page 143) chapter.

3.1. Getting Started with VNS 66

 VNS User Guide, Release 5.3.2

Port Address Translation

The PAT feature addresses local Internet offload use cases when the NSG is deployed as a WAN router for connectivity
to business Internet services. The feature enables internal user/application IP addresses to be mapped to a single
external routable address, as well as 1:1 mapping of external user/ application IP address to external routable address.
Provisioning includes:
• Enabling PAT at the domain/subnet level
• Creating Address Translation Pools
• Assigning Address Translation Pools to NSG or Network Port VLAN
• Configuring 1:1 NAT or 1:N PAT Maps
Refer to Port Address Translation (page 255) chapter.

PAT to Overlay

Enables the NSG with a distributed destination based PAT function from a branch domain to a local or remote shared
domain. Also supports NAT 1:1 mapping for specific hosts with routing policies to assign preferences per NSG.
Refer to PAT to Overlay (page 277) chapter.

BiDirectional NAT

The capability supports extranet scenarios where addressing might overlap on customer and provider networks or to
anchor traffic to a specific link/device. Bi-directional NAT enables performing source NAT (SNAT) and destination
NAT (DNAT) on a packet simultaneously between a customer and a provider domain.
Refer to BiDirectional NAT (page 287) chapter.

Application-Aware Routing

Enables policy-driven application performance management in a multi-path network. Specifically, AAR enables capa-
bilities such as application discovery, network performance measurement, and intelligent path selection for application
traffic based on path performance.
Refer to Application-Aware Routing (page 446) chapter.

NSG Border Router

NSG Border Router (NSG-BR) feature enables a secure demarcation point between the WAN and the DC underlay
networks. It provides seamless connectivity between NSG and non IPsec-capable endpoints (e.g. VRS, 7750 SR) by
linking domains, while maintaining a unified policy.
Refer to NSG Border Router (page 178) chapter.

NSG Underlay Border Router

NSG Underlay Border Router (NSG-UBR) feature enables seamless connectivity between NSGs in disjoint underlay
networks.
Refer to NSG Underlay Border Router (page 165) chapter.

3.1. Getting Started with VNS 67

 VNS User Guide, Release 5.3.2

Egress QoS

The QoS feature of VNS is the capability to control the rate of traffic egressing the NSG ports. The feature enables traf-
fic classification, prioritization and shaping to ensure that performance sensitive applications are prioritized, fairness
is maintained and/or packet-loss is minimized. Provisioning includes
• Defining Rate Limiters (CSPRoot function)
• Creating QoS Policies and associating Rate Limiters (CSPRoot function)
• Edit existing QoS Policies (Enterprise Administrator)
In addition Egress QoS Statistics can also be viewed for monitoring purposes.
Refer to Egress QoS (page 307) chapter.

IPsec VPNs

VNS supports two flavors of IPsec for traffic encryption:

• NSG to NSG traffic is encrypted using group-key based IPsec.
• NSG to 3rd party IKE Gateways is encrypted using IKE-based IPsec. The supported topology is hub and spoke,
with the NSG as either a hub or spoke.In 4.0.R1, only topologies where NSG is a spoke will be supported.
The provisioning for Group Key IPsec includes configuring the security related attributes at the CSPRoot and Enter-
prise level. Refer to Group Key IPsec (page 403) for detailed workflow.
IKE IPsec provisioning only has an enterprise scope. Refer to IKE IPsec (page 421) chapter.

Access Resiliency

The feature enables NSGs to be deployed as a redundant pair enabling HA configurations, protecting against link and
device failures.
Refer to Access Resiliency (page 242) chapter.

BGPv4 on NSG

Support for BGPv4 control plane enables the NSG to establish BGP peering with upstream PEs and with downstream
customer routers, and learn and advertise routes between the LAN and WAN subnets.
Refer to BGPv4 on NSG (page 325) chapter.

Secondary IP on NSG

Secondary IP support enables the NSG to have private and/or non-globally routable link-local interfaces between CE-
PE, thus saving the operator IP space and allowing a simplified provisioning model for PE by using the same interface
IP address on all CPE devices.
Refer to Secondary IP on NSG (page 155) chapter.

3.1. Getting Started with VNS 68

 VNS User Guide, Release 5.3.2

Service Chaining

Service chaining allows branch locations connectivity to datacenter-based service functions (e.g. firewall or load
balancer). Service chains may be provisioned using:
• Static routes on the domain instance level to redirect traffic (refer to the Nuage VSP User Guide)
• Advanced Forwarding ACLs (refer to the Nuage VSP User Guide)
Refer to Service Chaining in VNS (page 295) for details.

DHCP Relay

Enables DHCP Relay from NSGs to 3rd party DHCP servers behind a central NSG vPort (host/bridge). DHCP clients
are behind other NSGs in branch locations.
The Enterprise Administrator workflow for the DHCP Relay configuration is as follows:

1. Navigate to Networks -> Layer 3 Domains -> My L3 Domains (for existing instances). Otherwise instantiate
a domain instance from a template.

2. Once the instance is selected, select the settings icon in the far right panel.
3. Select the DHCP Behavior attribute and from the list of available options select DHCP Overlay Relayed.
4. Specify a primary and optionally a secondary DHCP server.

Note:
• The DHCP Behavior can only be modified if there are no vPorts attached in that domain.
• At least one DHCP server must be configured when DHCP Overlay Relayed option is selected.

5. Select each subnet in which the DHCP clients and DHCP server will reside and repeat the next step for each
selected subnet.

6. Select the settings icon in the far right panel.

7. Select the DHCP Relay Status attribute by scrolling in the panel. The attribute must be:
• enabled for each subnet in which the DHCP clients are present.
• disabled (default) for the subnet in which the DHCP server is present.
8. The following whitelisted command can be used to check the DHCP server configuration:

# ovs-appctl evpn/show alubr0 <evpn-id>

evpn_id: 1404641129 gen_id: 0x3 vni_id: 0x82ea2b ref_cnt: 2
mode: L3_MODE arp_proxy: DISABLED aging_period: 300
pat_enabled:DISABLED default_action:drop dhcp_enabled:ENABLED dhcp_relay:ENABLED
˓→dhcp_pool: DISABLED

resiliency: DISABLED l2_encryption:DISABLED

subnet: 20.0.2.0 mask: 255.255.255.0 gw: 20.0.2.1 gw_mac: 68:54:ed:00:24:20

dhcp servers: 20.0.0.100 20.0.0.150mac_count: 0 cookie: 371720192

port-name id refcnt flood IP subnet GW

port4.2 86 11 yes 0.0.0.0 0.0.0.0 0.0.0.0
ep1ltep-82ea2b 91 0 no

3.1. Getting Started with VNS 69

 VNS User Guide, Release 5.3.2

Vxlan Neighbors: 0
IPSec Neighbors: 0

3.1. Getting Started with VNS 70

 VNS User Guide, Release 5.3.2

3.2 Bootstrapping

• Overview (page 71)

• Pre-bootstrap NSG image upgrades (page 72)
– Performing a pre-bootstrap NSG image upgrade (page 73)
– Troubleshooting (page 73)
• Auto-Bootstrapping (page 73)
– LED Indicators during Auto-Bootstrapping (page 76)
– Considerations for bootstrapping over LTE uplink (page 76)
• One/Two-Factor Bootstrapping (Installer Workflow) (page 77)
– Prerequisites (page 77)
– WAN/LAN and Installer Ports on NSG (page 77)
– Installer Instructions (page 77)
– Viewing LTE information via Pre-bootstrap Status Page (page 84)
– Manual Configuration (page 85)
• CLI Based Bootstrapping (page 90)
– CLI Bootstrapping Examples (page 91)
• Pre-Bootstrap: Status Webpage (page 91)
• Pre-Bootstrap: Enable SSH (page 92)
– Enabling SSH Examples (page 93)
• Pre-Bootstrap: NSG WAN Checks (page 93)
– Sample uplink configuration (page 94)
• Pre-Bootstrap: NSG info (page 94)
• Pre-Bootstrap and Post-Bootstrap: NSG log collection (page 95)
• VSC CLI Show Command Examples (page 97)

3.2.1 Overview

The bootstrapping solution is a key part of Nuage VNS deployment that enables Nuage NSG end-points to securely
connect to Nuage VSCs (centrally hosted), download their service configuration, and connect end-users to their ser-
vices based on pre-defined policies with minimal user interaction. Several bootstrapping models are supported to
address the varying levels of customer site and installer trustworthiness as follows:
• Bootstrapping using one/two-factor installer authentication, when the NSG and installer are un-
trusted.
• Auto-Bootstrapping, when the NSG is deemed as physically secure and installer is trusted.

Note: The automatic bootstrapping capability is in line with existing industry solutions referred to as zero
touch or zero factor installation of CPE devices. However, Nuage enhances the capability for enterprises

3.2. Bootstrapping 71
 VNS User Guide, Release 5.3.2

that are mindful of security, and offers additional options for enterprises that want to maintain visibility
and control over CPE devices before allowing them to be onboarded.

The bootstrapping process:

• Is transport infrastructure independent. In other words the same process is invoked regardless of the underlying
transport being used to connect NSGs to the operator’s network.
• Does not require pre-configuration of NSGs prior to being dispatched to the site. Thus, the operator is not
required to have access to the appliance prior to it being commissioned.
• Allows a non-technical end-user (referred to as installer) to receive and commission the NSG. The installer does
not require special training or skill level for commissioning the NSG for any bootstrapping model.
For bootstrapping, it is assumed that:
• Nuage VNS software components have been installed and configured for connectivity per the procedure de-
scribed in the Nuage VNS Install Guide, and
• Infrastructure provisioning and any pre-staging required for bootstrapping has been completed as described in
Getting Started with VNS (page 18) chapter.

Attention: For LTE uplinks, a supported LTE device must be staged and shipped to the site by the Enterprise
administrator or service provider. For details, refer to LTE Staging and Operations Requirements (page 204).

Note: During bootstrapping, HTTPS communication between NSG and Utility VM utilizes TCP Ports 12443 and
11443. The procedure for modifying these ports, post installation of the VSD and Utility VM, is described in the Port
Modification for NSG Bootstrapping section of the VNS Install Guide.

3.2.2 Pre-bootstrap NSG image upgrades

Prior to bootstrapping and connecting the NSG to the VSD, you can upgrade the currently running software image
to another version using CLI. For example, you can upgrade the NSG from the gold image to allow the use of boot-
strapping features that are not available in the factory-default load, or even change to an earlier release. Pre-bootstrap
upgrades from CLI allow for the following scenarios:
• Upgrade from an image file on a USB drive.
• Upgrade from an image file that is locally available on the NSG (for example, after downloading from a remote
source using scp or wget).
Pre-bootstrap NSG image upgrades are performed with the nuage-operation -o UPGRADE command as the
Nuage user. See CLI-Based NSG Operations (page 106) for more information about command syntax and options.
Consider the following statements before performing a pre-bootstrap NSG upgrade:
• If performing the upgrade using a USB drive, do not plug the USB drive into the NSG until it has booted.
• Only 1 image .tar file can be present on the USB drive, otherwise the upgrade attempt will terminate with an
error message before starting.
• The USB drive must not be formatted in NTFS (EXT2, 3, 4, Fat32, and XFS are supported).
• When upgrading from a locally-stored .tar file, ensure that the file and checksum are stored in the /tmp directory
so that the Nuage user has access to the files.

3.2. Bootstrapping 72
 VNS User Guide, Release 5.3.2

• SSH must be enabled on the NSG if secure remote access is required for the upgrade. See Pre-Bootstrap:
Enable SSH (page 92).
• During the upgrade, you must ensure that no other operations are executed or secondary connections to the NSG
are active. Failing to do so can render the NSG inoperative.

Note: Pre-bootstrap NSG image upgrades require 100 MB of available RAM and 1 GB of available storage.

Note: The NSG-C does not have enough available RAM to run both the upgrade and MD5 checksum verification.
You must manually verify the MD5 checksum and execute the upgrade using the --ignoremd5 option.

Performing a pre-bootstrap NSG image upgrade

1. If you are performing the upgrade using locally available files, transfer the files to the /tmp directory on the
NSG.
2. If you are performing the upgrade using a USB drive, plug in the USB drive after powering on the NSG.
3. As the Nuage user, execute the sudo nuage-operation -o UPGRADE command with the following
syntax depending on the use case:
• sudo nuage-operation -o UPGRADE - run this command when upgrading using a USB
drive.
• sudo nuage-operation -o UPGRADE -f filename - run this command when upgrad-
ing using a locally available .tar file (for filename, enter the path to the .tar file).
You can add the --clean option to the end of the command to remove the upgrade-related files following
a successful upgrade (for example, if the .tar file and checksum were not located in /tmp).
4. Monitor the upgrade progress.

Troubleshooting

Insufficient memory to perform the upgrade operation - Ensure that 100 MB of RAM is available on the NSG to run
the upgrade. In the case of the NSG-C, the upgrade command must be run with the --ignoremd5 option to avoid
the insufficient memory error.

3.2.3 Auto-Bootstrapping

Auto-bootstrapping allows an NSG to be onboarded with minimal installer support. Nuage solution supports the
following auto-bootstrapping workflows to address various levels of security:
• A requesting NSG may be auto-assigned and auto-approved for bootstrapping which assumes that the device
and enterprise is completed trusted. This is equivalent to what industry refers to as zero factor installation.
• A requesting NSG may be auto-assigned to an enterprise, but requires an administrator to approve the bootstrap-
ping.
• A requesting NSG requires a CSPRoot to assign it to an enterprise and an enterprise administrator (or CSPRoot)
to approve the bootstrapping.

3.2. Bootstrapping 73
 VNS User Guide, Release 5.3.2

Fig. 3.41: Auto-Bootstrapping Process Flow

Auto-bootstrapping process involves several phases as shown in the figure (physical NSG form factor is assumed for
the illustration), specifically:
1. Pre-staging:
CSPRoot or Enterprise admin:
• Creates a userdata file which contains the necessary information that the NSG must send to the VSD
when it initiates an auto-bootstrapping request. Refer to CSPRoot workflow (page 45) and Enterprise
admin workflow (page 61) for creating userdata.
• Ships the userdata file (along with the NSG) on a USB (for physical NSG form factor) or as CDROM
(for the virtual NSG form factor).
2. Staging at Site (Installer Workflow):
The workflow described below assumes a physical NSG form factor. For a virtual form factor, the VM
boot-up sequence will mount the CDROM to read the userdata and the remaining workflow will be iden-
tical.
Installer:
• Receives the NSG and USB.
• Inserts the USB and powers up the NSG.
• Connects WAN uplink(s) to the configured Ethernet port(s) and/or inserts the LTE device (page 204)
(for the LTE uplink). It is expected that the Ethernet port information will be communicated to the
installer by the administrator.
NSG:
• Mounts the USB and reads the userdata file.
• NSG sends the data to the VSD along with a request for bootstrapping.
• The request will be either:

3.2. Bootstrapping 74
 VNS User Guide, Release 5.3.2

– Denied if the integrity check fails for the userdata being presented (this is to ensure that userdata
is not tampered), or
– Registered in pending state for review by CSPRoot (page 47) if there is no enterprise context or
review by Enterprise Admin (page 61) if there is enterprise context.

Note:
1. Once the NSG reads the userdata and is pending bootstrap approval, then the only way to delete
the userdata is to reset the NSG.
2. If the NSG is pending bootstrap approval, and if a USB with another set of userdata is inserted
and the NSG is rebooted, then a new auto-bootstrap request will be initiated. Thus, there will
be two pending requests on the VSD for the same NSG.

3. Review Pending Bootstrap Requests

When an NSG initiates an auto-bootstrapping request to the VSD, it is registered as a pending bootstrap
request at the CSPRoot level and/or at the Enterprise level.
The request registers in one of the following states:
• Unassigned: Pending CSPRoot review for assignment to enterprise.
• Assigned: Requesting NSG is auto-assigned to an enterprise as a result of a match and is pending
admin review for assignment to an instance.
• Approved: Requesting NSG is auto-assigned to an enterprise and an instance (auto-approved), and
bootstrapping is initiated on the NSG without any additional enterprise approval.
Refer to CSPRoot review workflow (page 47) and Enterprise Admin review workflow (page 61) for details.

Note: A request may also be denied if the userdata presented to the VSD fails integrity verification, and
in this case the request is not registered or presented on the VSD.

4. Clean Up Pending Bootstrap Requests

As part of the VNS configuration, the CSPRoot configures two timers that are used in the auto-bootstrapping process.
These are set in System Configuration by navigating to Settings -> (in the VNS Bootstrap Requests section):
• Bootstrap requests cleanup timer specified in minutes. Minimum is 1 minute and maximum is 10080 minutes
(7 days). Default is 1440 minutes (1 day).
• NSG request time interval specified in seconds. Minimum is 15 seconds, maximum is 240 seconds. Default is
30 seconds.

Fig. 3.42: Auto-Bootstrapping Timers

The Bootstrap requests cleanup timer is used to clean-up the pending request displays on the VSD for requests in
states unassigned, assigned, and denied. A pending request that is approved is cleaned up immediately.

3.2. Bootstrapping 75
 VNS User Guide, Release 5.3.2

The NSG request time interval value (say time-T) is downloaded to the NSG as part of the auto-bootstrap initializa-
tion request process. The NSG polls the VSD every time-T seconds to get the status of its pending bootstrap request.
The polling by the NSG is also treated by the VSD as a keepalive from the NSG.
In the event an NSG is powered down or loses connectivity after the initial auto-bootstrap request, and the VSD stops
receiving the keepalives, then the VSD removes the pending request after the Bootstrap requests cleanup timer
expires, and assumes that the NSG is out of service.
If the NSG gets re-connected at a later time, then in order to re-initiate the auto-bootstrap, the NSG must be rebooted.

LED Indicators during Auto-Bootstrapping

The NSG-E, NSG-E200, NSG-E300, NSG-X, and NSG-X200 appliances have LED indicators that provide progress
status during auto-bootstrapping as follows:
NSG-E, NSG-E200, NSG-E300:
• Blinking red: Corrupt userdata file on USB or file not found.
• Solid red: WAN check failed or proxy not reachable.
• Blinking green and red: WAN checks passed and proxy Reachable, but request rejected by VSD or VSD is not
reachable.
• Blinking green: auto-bootstrap request registered successfully with the VSD and is pending approval by either
CSPRoot/Admin.
• Solid green: NSG Bootstrapped successfully
NSG-X, NSG-X200:
• Blinking green: NSG has booted up but there is no USB (hence no auto-bootstrapping), and waiting for installer
to do one-factor or two-factor bootstrap
• Solid red: WAN check failed or proxy not reachable
• Alternate blinking green and red: WAN check passed, proxy is reachable but auto-bootstrap request rejected by
VSD or VSD not reachable
• Simultaneous blinking green and red: auto-bootstrap request registered successfully with the VSD and is pend-
ing approval by either CSPRoot/Admin
• Solid green: NSG Bootstrapped successfully

Considerations for bootstrapping over LTE uplink

Please note the following information:

• During bootstrapping over LTE uplink, integrated LTE devices and LTE dongles must not be plugged into the
NSG at the same time.
• Bootstrapping from gold image over LTE uplink is not currently supported for NSG platforms E206/306.
• Bootstrapping over a single LTE uplink is supported.
• In cases where users must switch network carriers for LTE uplinks, it is recommended to revoke the NSG, power
down the NSG, switch the SIM, and then re-bootstrap the NSG.

3.2. Bootstrapping 76
 VNS User Guide, Release 5.3.2

3.2.4 One/Two-Factor Bootstrapping (Installer Workflow)

Prerequisites

This workflow assumes that initial installer notification is done via email (e.g. mail server consuming VSD event
notification) and 2nd factor authentication via SMS (e.g. text to SMS gateway). A customer may opt to not configure
the 2nd factor.
1. The installer should have received an email with a link for registration, and should have the email ready for use
in the bootstrapping process.
2. The installer should also have information about which NSG Ethernet ports must be used for WAN uplink
connection(s) (single or dual uplinks).
3. The installer’s laptop should have a DHCP client running so that it can receive an IP address from the NSG.
4. It is recommended that Wi-Fi be disabled on the installer’s laptop to reduce the likelihood of IP address allocation
from the Wi-Fi network.
5. As part of the NSG bootstrapping process, the WAN interface on the NSG must be successfully assigned an IP
address either dynamically (for DHCP connections), statically, via PPPoE, or LTE.
6. For LTE uplink, installer should have received a supported LTE device (page 204).
7. If the NSG includes an installer-managed PPPoE connection and is configured with a dual-uplink, bootstrapping
may succeed through the non-PPPoE uplink even if the credentials on the PPPoE uplink are incorrect. If this
scenario occurs, the VSD raises a VRS_DISCONNECTED_FROM_CONTROLLER alarm. To avoid this scenario,
the installer must bootstrap the NSG through the PPPoE-configured port only. When PPPoE configuration is
successful, the installer should then plug in the non-PPPoE port and proceed with bootstrapping.

WAN/LAN and Installer Ports on NSG

Refer to the “Port Lists” section in the VSP Install Guide for information about the WAN/LAN ports to be used for
installation.
The CSPRoot or Enterprise administrator configures the network and access ports.

Installer Instructions

1. For bootstrapping an NSG-V form factor, prepare installer’s laptop as follows:

• Edit the /etc/hosts file (requires root privileges) to resolve “registration.nsg” to the uplink IP address of the
NSG-V VM, or
• Open a browser, copy the registration link in the browser, and replace the string “registration.nsg” with the
uplink IP address of the NSG-V VM
2. For physical form factor of the NSG, specifically NSG-C/E/X, proceed as follows:
• Connect the NSG to your WAN network using the appropriate Ethernet port(s) or USB interface for
LTE.
• After the NSG is powered up, connect the installer’s laptop to the installer port (page 77) on the
NSG. NSG allocates an IP address to the laptop. This is required for bootstrapping.
The next steps are common to virtual and physical form factors:
3. Click on the registration link provided in the e-mail which opens a captive portal presented by the web server
running on the NSG.

3.2. Bootstrapping 77
 VNS User Guide, Release 5.3.2

Note: For the LTE uplink scenario, the installer can view the LTE information on the device by accessing the pre-
bootstrap status page (page 84).

4. If the NSG is configured with a PPPoE uplink connection that is installer-managed, PPPoE credentials must
be entered before bootstrapping can proceed. Specify the Username and Password before clicking the Apply
Manual Configuration button.
See PPPoE Mode Parameters (page 56) for more information on installed-managed PPPoE configuration.

Note: If the bootstrapping process is terminated for any reason after entering PPPoE credentials, the NSG
should be rebooted before another registration link is requested. Rebooting the NSG restarts the PPPoE
service and ensures that credential verification is performed during the second bootstrapping attempt.

Fig. 3.43: Installer-managed PPPoE uplink connection configuration.

5. The NSG performs automated self-tests to check that:

• IP@ is configured on uplink(s) port(s) of NSG
• Gateway is reachable - check implemented using ARP request
• DNS resolution test for proxy FQDN
• Proxy reachability test via TCP to bootstrapping end-point configured on proxy on port 11443.

3.2. Bootstrapping 78
 VNS User Guide, Release 5.3.2

Fig. 3.44: System Connection Progress

6. If the system connectivity test fails for any reason (e.g. IP@ is not present on the WAN uplink port because
there is no DHCP, or static IP@ sent in registration link is misconfigured, or PPPoE credentials are wrong), then
bootstrapping will not proceed, and the following error will be displayed.

Fig. 3.45: WAN Check Failure Error

The installer will have to contact the Enterprise Administrator and request a new registration link to
address the proxy reachability issues. If the NSG is pre-4.0.R3, then installer is presented with a manual
configuration (page 85) option.

3.2. Bootstrapping 79
 VNS User Guide, Release 5.3.2

Note: The detailed activation logs displayed in the Advanced logs can sometimes provide more troubleshooting
information. In order to collect all the local NSG logs from the browser, navigate to http://registration.nsg
and click on the top right button Download NSG logs.

7. Once the proxy is resolved and reachable, device is ready to be activated. If two-factor authentication is enabled,
the following page is presented.

Fig. 3.46: Activate NSG using Two-factor Authentication

The six-digit activation code is sent via SMS to the mobile phone specified in the VSD user profile. The
code expires ten minutes after being sent. The activation code length and expiration time can both be
configured in the VSD system configuration settings.
If the SMS is not received in a reasonable time, click on the Request a new code button to request a new
code. A new code is sent to the specified mobile phone and the old code immediately becomes invalid.
If the code is entered incorrectly, the following page is presented. Reenter the code or request a new one.

3.2. Bootstrapping 80
 VNS User Guide, Release 5.3.2

Fig. 3.47: Request a New Activation Code

Note: Do not reload the activation code request webpage. This causes a new code to be sent and the old
code to become invalid.

If the activation code entered has expired, the following page is presented. Click on the Request a new code button
to request a new code.

3.2. Bootstrapping 81
 VNS User Guide, Release 5.3.2

Fig. 3.48: Activation Code Expiry

If two-factor authentication is not being used, the following page is presented.

Fig. 3.49: Activate NSG

Click on the Activate your device button to begin the bootstrapping process.
8. Once the bootstrap process completes successfully, the NSG is considered to be in active state.

3.2. Bootstrapping 82
 VNS User Guide, Release 5.3.2

Fig. 3.50: Bootstrap Completed Message

9. If the bootstrap process aborts unsuccessfully and the following page is presented, then the VSC infrastructure
profile information will have to be correctly entered by the CSPRoot.

Fig. 3.51: VSC Configuration Error Message

The CSPRoot will have to follow these steps:

• Enter the VSC profile information as described in Create and Assign Infrastructure VSC Profile
(page 37)
• Manually deactivate the NSG by following the Revoke Gateway Access (page 105) procedure.
– Notify the installer for re-bootstrapping the device by clicking on the Send Activation button.
10. If bootstrapping fails, the following page is presented. Click on the Download NSG logs button to download
an archive with files detailing events the the NSG logged during bootstrapping. See Pre-Bootstrap and Post-
Bootstrap: NSG log collection (page 95) for more information about NSG log collection.

3.2. Bootstrapping 83
 VNS User Guide, Release 5.3.2

11. If the NSG is already bootstrapped, the following page is presented. If the NSG needs to be bootstrapped again,
you must physically reset it first. If the error persists, you must revoke the NSG from the VSD and from the
NSG CLI.

Viewing LTE information via Pre-bootstrap Status Page

When an LTE uplink is configured, the installer can enter the URL http://registration.nsg (or the corre-
sponding IP address) in the browser, which will open the Pre-Bootstrapping Status page.
The page will contain information related to auto-bootstrapping (if present), IP addresses for Uplinks, LTE devices
connected, and the LTE Information gathered by the device if it has connected to a provider.

3.2. Bootstrapping 84
 VNS User Guide, Release 5.3.2

See Pre-Bootstrap: Status Webpage (page 91) for more information about the NSG status webpage.

Manual Configuration

Manual configuration allows the installer to configure connection parameters manually during NSG bootstrapping.
Manual configuration may be required if WAN checks fail during bootstrapping, depending on the release and config-
uration details of the bootstrapped NSG and the VSP.
• If the bootstrapped NSG is release 4.0 R2 or earlier and WAN checks fail, manual configuration is required.
• If the bootstrapped NSG is release 4.0 R3 or later and the VSP is release 4.0 R3 or later, the NSG instance may
be configured with the Connection type Any. If the NSG instance is configured in the VSD with the Connection
type Any and WAN checks fail, manual configuration is required.
• If the bootstrapped NSG is release 4.0 R3 or later and the VSP is release 5.0.1 or later, the Connection type Any
cannot be configured via the VSD. However, the value may still exist in NSG instances that were configured
prior to the upgrade to VSP release 5.0.1 or later. If the NSG instance has Connection type Any and WAN
checks fail, manual configuration is required.

Note: If the Connection type Any still exists in legacy NSG instances in VSP release 5.0.1 or later, the NSG uplink
must be reconfigured as Static, Dynamic, or PPPoE. See Define Uplink Connections (page 54) for more information
about configuring uplink connections.

Perform the following steps to proceed with manual configuration.

3.2. Bootstrapping 85
 VNS User Guide, Release 5.3.2

Fig. 3.52: Manual Configuration of WAN Uplink

• From the Mode dropdown, select PPPoE or Static Configuration.

Follow the workflow for PPPoE Configuration or Static Configuration described below.
PPPoE Configuration

3.2. Bootstrapping 86
 VNS User Guide, Release 5.3.2

Fig. 3.53: PPPoE Configuration

For PPPoE, enter the following:

• ISP provided username and password. If the connection is successful, an IP address will get assigned to the
network port and the device will be ready for activation. In the event the IP address assignment fails, then the
same manual configuration window will be presented with an error code.

Note:
• PPPoE configuration is persistent across NSG reboots and upgrades.
• A PPPoE connection entails an 8 Byte overhead. Also, the MTU of uplink port will drive Maximum Segment
Size (MSS) clamping to adjust the MSS based on port’s MTU

If there are two WAN uplinks, then check the Enable Dual Uplink box, so that the page expands to include the
attributes for the second uplink.

3.2. Bootstrapping 87
 VNS User Guide, Release 5.3.2

Fig. 3.54: Manual Configuration (PPPoE and Dual WAN Uplinks)

Static Configuration

3.2. Bootstrapping 88
 VNS User Guide, Release 5.3.2

Fig. 3.55: Static Configuration of WAN Uplink

Enter the following:

• IP Address entered using IPv4 IP@ notation
• Netmask entered using full netmask notation (e.g. 255.255.255.0)
• Gateway IP address entered using IPv4 IP@ notation
• DNS Server IP address using IPv4 IP@ notation
If there are two WAN uplinks, then check the Enable Dual Uplink box, so that the page expands to include the
attributes for the second uplink.

3.2. Bootstrapping 89
 VNS User Guide, Release 5.3.2

Fig. 3.56: Static Configuration (Dual WAN Uplinks)

3.2.5 CLI Based Bootstrapping

NSG CLI can also be used to initiate bootstrapping. CLI bootstrapping can be performed using either a URL or a yaml
file. A skeleton yaml file can be generated to use as a template for bootstrapping file creation.
Limitation:
• CLI Based Bootstrapping cannot be used for Two-Factor Bootstrapping. It is only supported for Auto-
bootstrapping or One-Factor bootstrapping.
Login to the NSG as the “nuage” user, and execute the sudo nuage-bootstrap-cli command to initiate CLI
bootstrapping and specify either a URL or file that contains the required bootstrapping parameters. The following
parameters are available for the command:
• -h, --help - Show the help message and exits.
• -u URL, --url URL — Provide the Bootstrap URL.
• -f FILE, --file FILE — Provide yaml file required for auto bootstrapping.
• -g FILENAME, --gen-sample FILENAME — Generates a skeleton yaml file under /home/user/nuage.
This file can be used to create a bootstrapping file.

Note: When bootstrapping with a URL, copy and paste the full registration link that is provided. There is no need to

3.2. Bootstrapping 90
 VNS User Guide, Release 5.3.2

include an IP address in the bootstrapping URL.

CLI Bootstrapping Examples

The following is a sample execution of CLI bootstrapping using a URL:

-bash-4.2$ sudo nuage-bootstrap-cli -u "http://registration.nsg?data=Sa1Mp3Le5cOd6E"

Required configuration for uplink(s) completed.
Checking if uplink ports are set.
IP information detected for both uplink ports.
Checking DNS resolution for the Proxy/Util system.

The following is a sample execution of CLI bootstrapping using a file:

-bash-4.2$ sudo nuage-bootstrap-cli -f file.yaml

Starting Auto-Bootstrap with file.yaml

The following is the skeleton yaml file generated with the -g parameter:

-bash-4.2$ sudo nuage-bootstrap-cli -g template

Generated sample userdata is /home/user/nuage/template.yaml
-bash-4.2$ cat /home/user/nuage/template.yaml
#This is just a sample, actual file needs to be generated from VSD
#cloud-config
nuage_nsg:
proxyFQDN: <Proxy FQDN>
enterpriseID: <UUID of the Enterprise in VSD>
NSGatewayID: <UUID of the NSG in VSD>
NSGType: <ANY | NSG-E | NSG-V>
# List of uplinks, a sample static uplink configuration is shown below
uplinks:
- name: port1
v4:
mode: static
static:
gw: 10.10.10.21
ip: 10.10.10.25
dns: 10.10.10.120
mask: 255.255.255.240
signature: <Signature of this yaml as generated by VSD>

Note: Refer to the ISO or YAML bootstrapping file from the VSD for the information required for this YAML file.

3.2.6 Pre-Bootstrap: Status Webpage

An NSG status webpage is available from the LAN access ports prior to the NSG bootstrapping, at this location:
http://registration.nsg/ or http://172.16.1.1/
The Web Page presents real time information on:
• Auto-Bootstrap Status - In case Auto-Bootstrapping method is used.
– Auto-Bootstrap Info

3.2. Bootstrapping 91
 VNS User Guide, Release 5.3.2

– WAN Check Status

– Details
• Mobile Connection Information - Displays details on LTE Status.
• USB Devices Detected - Displays details for detected LTE devices. Displays ICCID/IMSI for integrated LTE
devices.
• IP Information -Current ports configuration.
• Storage Information - Current used and free space for /dev/sda.
• Memory Information - Current used and free memory.
• SSH Status and Configuration - If checked, the SSH service is currently running. If unchecked, the SSH service
is currently stopped.

Fig. 3.57: NSG Status Web Page

3.2.7 Pre-Bootstrap: Enable SSH

By default, the SSH service is not started on a pre-bootstrap NSG. Enabling the SSH service for remote access can be
done in two ways: 1. Through the NSG console/CLI 2. Using the NSG Status Web Page
Login to the NSG as the “nuage” user or the equivalent default user, and execute the nuage-ssh-config command
using the following options to start the SSH service:
• -h, --help - Show the help message and exits.
• -a ACTION, --action ACTION - Possible values are USE_PWD or USE_SSHKEY.
• -k SSHKEY, --sshkey SSHKEY - Authorized Public SSH Key for the Nuage user.

Note: Post bootstrap, the SSH service is configured on the NSG based on the VSD Infrastructure Access Profile

Note: Custom SSH configuration is not persistent: it returns to default values after a NSG reboot.

3.2. Bootstrapping 92
 VNS User Guide, Release 5.3.2

Enabling SSH Examples

-bash-4.2$ nuage-ssh-config -a USE_PWD

Request to Enable Password based Authentication has been received.
SSH Service Started.
Authentication mode set to Password only.

-bash-4.2$ nuage-ssh-config -a USE_SSHKEY -k <YOUR_SSH_KEY>

Request to Enable Key based Authentication has been received.
SSH Service Started.
Authentication mode set to SSH Key only.

3.2.8 Pre-Bootstrap: NSG WAN Checks

An uplink configuration script can be executed via CLI to allow you to configure a single WAN uplink prior to
bootstrapping. This allows the user to perform WAN connectivity checks without having to bootstrap the NSG.

Note: The uplink configuration script must be executed using sudo.

Execute the nuage-uplink-config command to configure the uplink. The following parameters are available
for the command:
• -m CONNECTIONMODE — Defines the uplink type. Specify “STATIC” or “DYNAMIC”.
• -if INTERFACE — Defines an interface, such as a port number.
• -ip IPADDRESS — Defines an IPv4 address to be configured on the interface.
• -net NETMASK — Defines a netmask to be configured on the interface.
• -gw GATEWAY — Defines the default gateway to be configured on the interface.
• -dns DNSSERVER — Defines a DNS server to be configured on the interface.
• -mtu PORTMTU — Defines a port MTU value to be configured on the interface. The range is 68-9198.
• -speed PORTSPEED — Defines a port speed to be configured on the interface. Supported Ethernet speeds
are 10 (10Base-T), 100 (100Base-TX), 1000 (1000Base-T), 10000 (10GBase-X), and 0 (auto-negotiate).
The port MTU and speed can be defined during uplink configuration, or afterwards.
Once the WAN uplink is configured, the following LINUX commands can be executed to check WAN connectivity:
• Ping
• Tracepath
• Route
• NSLookup
• Dig
Consider the following when executing the uplink configuration script:
• With the exception of port MTU and speed, it is expected that all parameters required for uplink configuration
are defined with one execution of the command.
• The script checks if the NSG is already in a pre-bootstrapping state by checking if the merged config file contains
a valid gateway UUID.

3.2. Bootstrapping 93
 VNS User Guide, Release 5.3.2

• When the NSG is bootstrapped, the NSG receives the uplink configuration from the VSD. If the configuration
does not match with the configuration on the NSG, bootstrapping may fail. If the NSG is configured with a dual
uplink with only one mismatched configuration, bootstrapping should succeed.
• Uplink configuration is performed per uplink. If a dual uplink is required, the script must be executed twice.
• These configurations are not persisted on the NSG: once NSG is rebooted, the ports configuration are set back
to default values.

Sample uplink configuration

The following is a sample DHCP uplink configuration using the uplink configuration script.
-bash-4.2$ sudo /usr/bin/nuage-uplink-config -m DYNAMIC -if port2
Uplink Configuration Request Received for Interface port2
[{'v4': {'mode': 'dhcp'}, 'name': 'port2'}]
Uplink Configuration has been completed for Interface port2

The following is a sample static uplink configuration using the uplink configuration script.
-bash-4.2$ sudo /usr/bin/nuage-uplink-config -m STATIC -if port2 -ip 135.227.176.174 -
˓→net 255.255.254.0 -gw 135.227.176.1 -dns 1.1.1.1

Uplink Configuration Request Received for Interface port2

[{'v4': {'static': {'gw': '135.227.176.1', 'ip': '135.227.176.174', 'mask': '255.255.
˓→254.0', 'dns': '1.1.1.1'}, 'mode': 'static'}, 'name': 'port2'}]

Uplink Configuration has been completed for Interface port2

The following is a sample of port MTU configuration on an uplink that has already been configured.
-bash-4.2$ sudo /usr/bin/nuage-uplink-config -if port2 -mtu 1300
Done with MTU configuration on Interface port2

3.2.9 Pre-Bootstrap: NSG info

NSG info can be viewed by a Nuage user from the CLI pre-bootstrap. Run the nsginfo script to see the NSG type,
serial number, MAC address, UUID, TPM status and other information.
The TPMStatus parameter indicates whether TPM is enabled and operational on the NSG. The possible values are
as follows:
• 0 — TPM status is unknown. This value may be displayed for an NSG-V or a physical NSG without a TPM
chip.
• 1 — TPM is enabled, but not operational. This value may be displayed when TPM is enabled in BIOS, but
cannot be used.
• 2 — TPM is enabled and operational.
• 3 — TPM is disabled. This value may be displayed when TPM is detected on the device, but has been disabled
or not properly configured in BIOS.
The following is a sample output of the nsginfo script.
-bash-4.2$ nsginfo
{
"SKU": "3HE09966AARA01",
"MACAddress": "28:53:fd:22:11:f6",
"hostname": "localhost.localdomain",

3.2. Bootstrapping 94
 VNS User Guide, Release 5.3.2

"UUID": "Not Specified",

"family": "NSG-E",
"memoryTotal": "4035164 kB",
"serialNumber": "NS1543Q0049",
"TPMVersion": "1.2.3.19",
"productName": "7850 NSG-E",
"libraries": "cfengine-community-3.10.0-1.x86_64; openssl-tpm-engine-0.4.2-1.x86_
˓→64openssl-tpm-engine-0.4.2-1.i686; libnavl-4.5.0-nuage.el7.centos.x86_64;

˓→NetworkManager-1.0.0-16.git20150121.b4ea599c.el7_1.x86_64",

"BIOSReleaseDate": "06/18/2015",
"BIOSVersion": "5.6.5",
"NSGVersion": "Nuage NSG 0.0.0_NSGPR1591_be077e",
"TPMStatus": 1,
"CPUType": "Intel(R) Atom(TM) CPU C2358 @ 1.74GHz",
"CPUMHz": "1750.071",
"IPAddress": "134.127.276.34",
"CPUCacheSize": "1024 KB"
}

3.2.10 Pre-Bootstrap and Post-Bootstrap: NSG log collection

During NSG bootstrapping, each page in the bootstrapping captive portal includes a Show Advanced Log button. The
button expands or collapses a window that shows the logs generated by the NSG during bootstrapping. The window
provides a visualization of bootstrapping warnings, errors, and info messages as they are raised in realtime. The
advanced log window helps the installer provide more information to the support team if NSG bootstrapping fails.

An archive of NSG log files can be collected by the installer. These log files can be useful for troubleshooting if a
support team member needs advanced log details about a failed bootstrapping attempt. The log files can be generated
via two methods:
• Via the NSG bootstrapping captive portal prior to NSG bootstrapping.
• Via the CLI prior to or after NSG bootstrapping.

3.2. Bootstrapping 95
 VNS User Guide, Release 5.3.2

Logs can be generated on the NSG bootstrapping captive portal from the NSG Status Web page. Click on the Download
NSG Logs button to download the archive of logs.
Logs can be collected from the CLI prior to or after NSG bootstrapping. Using SSH or the console port, log in as a
nuage user and run the logexport script.
Once the logs are collected, they are saved to the archive at /tmp/logexport.tar.gz. The nuage user can
then copy the archive to a remote host using the scp tool. The files in /tmp/logexport.tar.gz are stored
temporarily and are deleted after one day.
When the NSG undergoes a graceful reboot or shutdown, the NSG copies the logexport.tar.gz file to /home/user/nuage.
The archive is overwritten with each reboot or shutdown, and deleted when the NSG is deactivated (revoked).
The following logs are collected in the archive:
• /home/nuage/nsginfo
• /var/log/openvswitch/nuageMon.log
• /var/log/openvswitch/nuage-rpc.log
• /var/log/openvswitch/nuage-SysMon.log
• /var/log/openvswitch/ovsdb-server.log
• /var/log/openvswitch/ovs-vswitchd.log
• /var/log/openvswitch/vrs-rgsyncd.log
• /var/log/openvswitch/nuage-ovs.log
• /var/log/openvswitch/nuage-nsg-ike-cfg.log
• /var/log/openvswitch/nuage-nsg-ike-cli.log
• /var/log/charon.log
• /var/log/openvswitch/nuage-dtls.log
• /var/log/openvswitch/bgpd.log
• /var/log/openvswitch/nuage-nsg-bgp-cfg.log
• /var/log/openvswitch/nuage_perfd.log
• /var/log/openvswitch/nuage-dpi.log
• /var/log/openvswitch/nuage-perfmon-config.log
• /var/log/openvswitch/nuage-qos-config.log
• /var/log/messages
• /var/log/sdvpn/agent.py.log
• /var/log/sdvpn/bootstrap.py.log
• /var/log/sdvpn/captiveportal.py.log
• /var/log/sdvpn/cfengine.log
• /var/log/sdvpn/deadTimer.py.log
• /var/log/sdvpn/dpdk.py.log
• /var/log/sdvpn/ifOperStatus.py.log
• /var/log/sdvpn/installerLED.py.log
• /var/log/sdvpn/iptables.log

3.2. Bootstrapping 96
 VNS User Guide, Release 5.3.2

• /var/log/sdvpn/lte.py.log
• /var/log/sdvpn/nsginfo.py.log
• /var/log/sdvpn/ssh_mgmt.py.log
• /var/log/sdvpn/systemUpgrade.py.log
• /var/log/sdvpn/tpm.py.log
• /var/log/sdvpn/webapi.py.log
• /var/log/sdvpn/zfbinit.py.log

3.2.11 VSC CLI Show Command Examples

The following commands provide visibility into PPPoE session information:

*A:vsc# pppoe-status
pppoe-status: Link is up and running on interface ppp0
31: ppp0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1492 qdisc pfifo_fast state
˓→UNKNOWN qlen 3

link/ppp
inet 10.10.1.2 peer 10.10.1.254/32 scope global ppp0
valid_lft forever preferred_lft forever

*A:vsc# systemctl status pppoe-client -l

pppoe-client.service - PPPoE client start service
Loaded: loaded (/usr/lib/systemd/system/pppoe-client.service; static)
Active: active (exited) since Fri 2016-04-15 22:43:15 UTC; 4 days ago
Process: 14457 ExecStart=/usr/sbin/pppoe-start (code=exited, status=0/SUCCESS)
Main PID: 14457 (code=exited, status=0/SUCCESS)
CGroup: /system.slice/pppoe-client.service
\u251c\u250014465 /bin/bash /sbin/pppoe-connect
\u251c\u250014492 /usr/sbin/pppd pty /usr/sbin/pppoe -p /var/run/pppoe-adsl.
˓→pid.pppoe -I port1 -T -U -m 1412 ipparam ppp0 linkname ppp0 noipdefault noauth
˓→default-asyncmap defaultroute hide-password nodetach usepeerdns mtu 1492 mru 1492

˓→noaccomp nodeflate nopcomp novj novjccomp user mk lcp-echo-interval 20 lcp-echo-

˓→failure 3

\u2514\u250014493 /usr/sbin/pppoe -p /var/run/pppoe-adsl.pid.pppoe -I port1 -

˓→T -U -m 1412

3.2. Bootstrapping 97
 VNS User Guide, Release 5.3.2

3.3 Gateway Operations

• Overview (page 99)

• NSG License Management (page 99)
• NSG System Information (page 100)
• Configuration Updates (page 101)
• Configuration Status (page 102)
• Proxy Reachability (page 102)
• NSG Security (page 103)
– Auto-Deactivation (page 103)
– Controllerless Operation (page 104)

* IPsec Key Considerations (page 104)

– Alarms for Gateway Operational States (page 105)
• On-Demand Gateway Deactivation (page 105)
– On-Demand Gateway Deactivation via CLI (page 106)
• CLI-Based NSG Operations (page 106)
– NSG image upgrade (pre-bootstrap) (page 107)
– Gateway Deactivation (page 107)
– Certificate Renewal (page 108)
– NSG Patch List (page 108)
– Export core dumps (page 108)
• Setting up Secure Access to NSG (page 108)
• Trusted Platform Module (TPM) Status (page 111)
• NSG Upgrade Profiles (page 113)
– NSG Upgrade Profiles Workflow (page 113)
• NSG Patching (page 114)
– NSG Patch Profile (page 114)
– NSG Patch Bundle (page 115)
– Detailed Patch Monitoring (page 115)

* Patch Status Scenarios (page 116)

– NSG Patch List (page 116)
– NSG Patch Profiles Workflow (page 116)
• Patch Operation Status Codes (page 117)
– Unknown Status (page 118)
– NSG Upgrade Statuses (page 118)

3.3. Gateway Operations 98

 VNS User Guide, Release 5.3.2

– NSG Patch Statuses (page 118)

• Datapath CLI commands (page 119)
– nuage-flow-top (page 119)
– ovs-appctl bridge/dump-conntracks-summary alubr0 (page 120)
– ovs-appctl bridge/dump-conntracks alubr0 (page 120)
– conntrack -L (page 121)
• CLI Output (page 122)

3.3.1 Overview

This chapter describes the NSG management functions related to NSG operations, specifically licensing, maintenance,
monitoring and security.

3.3.2 NSG License Management

The count of active (i.e. bootstrapped) NSG instances can be viewed in the VSD UI by navigating to Settings –> .
Only the active (i.e. bootstrapped) NSGs are counted. Non-Bootstrapped NSG instances do not count against license
count checks.

Fig. 3.58: NSG License Count

At the time of a bootstrapping request, the VSD provides immediate notification (via the proxy channel) of license
violation which results in the bootstrap request being rejected and an alarm being raised. Otherwise the bootstrap is
allowed to proceed. Anytime an NSG is deleted, the count is incremented.
Alarms are raised when:
• Number of bootstrapped NSGs exceeds 90% of the allowed count.
• There are 15 days to expiry of a license.

3.3. Gateway Operations 99

 VNS User Guide, Release 5.3.2

• License expires.
For detailed license and alarm management instructions, refer to the latest version of the Nuage VSP User Guide.

3.3.3 NSG System Information

Fingerprint information for a bootstrapped NSG can be viewed by navigating to the NSG instance and clicking on the
system information icon in the righmost panel.
The following are example displays for NSG-V, NSG-E, and NSG-AMI (on AWS) instances.

Fig. 3.59: System Information for NSG-V

Fig. 3.60: System Information for NSG-E

3.3. Gateway Operations 100

 VNS User Guide, Release 5.3.2

Fig. 3.61: System Information for NSG-AMI

In addition to the fingerprint information displayed above, the system information, for each form factor, also displays
configured SSH attributes. Refer to setting up secure access to NSG (page 108) for a more detailed description.

3.3.4 Configuration Updates

Typically the VSC facilitates infrastructure configuration updates from the VSD to be pushed to the NSGs on demand.
An NSG polls the VSD periodically (as defined by the synchronization parameter in its profile) for any infrastructure
configuration updates, shown as circled 1 in the figure below. However, on occasion the owner of an instance can
override the scheduled polling and force the NSG to apply configuration updates instantaneously by selecting Reload
Configuration from the gateway interface. At such time:
• VSD notifies VSC that NSG configuration has changed, see 2a in this example. There is an option to configure
TLS for XMPP for secure VSC-VSD communication. This requires that the XMPP server on the VSD is
configured to run in “allow” or “require” mode.
• VSC via “JSON-RPC” notifies NSG that it needs to retrieve the configuration from VSD, see 2b in this example.
• NSG uses “https” channel via proxy to download configuration from VSD, see 2c in the graphic.
Example

3.3. Gateway Operations 101

 VNS User Guide, Release 5.3.2

Fig. 3.62: NSG Configuration Management

3.3.5 Configuration Status

The following configuration status is displayed in the VSD UI as part of the Device Information panel of the selected
NSG instance (Infrastructure –> Network Service Gateways –> NSG instance –> ):
• Configuration Status
• Last Configuration Reload on which displays the actual time when either a reload configuration was applied
or configuration was updated at the sync window.

Fig. 3.63: Example Configuration Status

3.3.6 Proxy Reachability

The NSG must be able to resolve the Proxy FQDN to the IP address associated with the uplink port. The resolved IP
address is used to forward self-generated traffic (e.g. statistics/NTP/reload config) via the associated uplink port. The
assumption is that the IP address resolved by an upstream DNS Server (via an uplink port) is reachable via the same
underlay.
When an NSG has dual uplinks in disjoint underlays, then the DNS queries are sent to all DNS servers reachable via
each uplink. Flexible DNS reachability and resolution is one of the key requirements in SD-WAN deployments. The
DNS forwarder enhancement, introduced in 4.0.R8, enables the NSG to make forwarding decisions by accounting for
which DNS server and underlay provided the DNS resolution. Given that the DNS queries are sent to all DNS servers
reachable via an uplink, the first DNS server that responds to the DNS query will be honored, and the resolved IP will
be associated with the uplink port. If DNS resolution is successful on both uplinks, then the NSG selects Port 1 uplink

3.3. Gateway Operations 102

 VNS User Guide, Release 5.3.2

for forwarding traffic using the resolved IP on that port. If Port 1 uplink fails, then a new DNS query will trigger the
NSG to select Port 2 uplink for forwarding traffic.
The feature supports the following DNS scenarios in disjoint underlays:
• Single DNS server that has the same IP as the Proxy
• DNS servers in each underlay with overlapping IPs
• DNS servers in each underlay with distinct IPs

3.3.7 NSG Security

Security measures are built in to isolate and protect the integrity of an NSG in the event it loses control plane connec-
tions to the VSC(s).

Note: An NSG is considered to be disconnected from the VSC if any one of the control connections (OpenFlow,
JSON or DTLS) goes down and/or if the NSG does not receive a seed update when the remaining seed lifetime is less
than the seed regeneration interval (such as when the key server is down).

For such controllerless contingencies, an NSG may be configured to transition into different operational states ranging
from maintaining normal forwarding, or limited forwarding and/or no forwarding with an option of being revoked.
Refer to Infrastructure Gateway Profile (page 30) for workflow.

Fig. 3.64: NSG Operations Modes

The NSG operations modes are:

Auto-Deactivation

In auto-deactivation mode, the NSG stops forwarding all traffic and the auto-deactivation timer is started to allow the
NSG to restore connectivity to the VSC(s). Auto deactivation timer is applied as follows:
• When an NSG is disconnected from the VSD/VSC for longer than the duration specified. On expiry of the timer,
the NSG is considered to be untrusted and is not allowed to re-establish connection to the VSC. This is achieved
by revoking the NSG certificates and resetting the NSG to the pre-bootstrap state, which could take up to 90
minutes after expiry of the timer.

3.3. Gateway Operations 103

 VNS User Guide, Release 5.3.2

• If the NSG is powered down for a period longer than the duration specified, it will automatically reset itself to a
pre-bootstrap state once it is powered on. This operation is local to an NSG device.

Note: A bootstrapped NSG can also be deactivated on demand as described in Revoke Gateway Access (page 105).

Controllerless Operation

• Local and Remote: In this mode, the NSG maintains a normal mode of operation, and controllerless remote
duration timer is started to allow the NSG to restore connectivity to the VSC(s). During this time local and
remote flows are maintained. Upon expiry of the timer, NSG transitions to the Local Only mode.
• Local Only: In this mode, the NSG maintains limited forwarding, and controllerless local duration timer is
started to allow the NSG to restore connectivity to the VSC(s). During this time, only local routing is maintained
and all remote routes are removed. Upon expiry of the controllerless local timer, the local routes are also
removed.

Note:
• The NSG should not be viewed as a local LAN router. During controllerless operation, it will only maintain
routing between existing LAN hosts (within and between local subnets). Thus, in controllerless mode, if a new
host is added, it will be able to communicate to hosts within the same subnet but not to hosts across subnets.
• From a VSD perspective controllerless operation could also mean that the NSG is powered down. There is no
distinction being made with respect to the operations state.

IPsec Key Considerations

Cryptographic keys used for IPsec encryption and decryption on the NSG are dependent on the availability of active
Seed data from the Key Server. Seed received by the NSG from the keyserver via the VSC has a fixed lifetime when
it is considered valid for generating traffic keys (security associations).
In the event of a connectivity loss between NSG and VSC, or between VSC and VSD, the Seed may not be delivered
to the NSG which results in keys not being available for encrypting traffic and hence forwarding will stop.
During controllerless contingencies, the NSG utilizes a longer living disaster recovery (DR) Seed that is used to
encrypt IPsec traffic for continuity during the controllerless remote duration when normal forwarding is maintained.
The DR Seed lifetime is the maximum of all the configured controllerless remote duration in the enterprise.
Thus, if the NSG is in controllerless local and remote mode, and the key generation Seed lifetime expires, then the DR
Seed is used for the remaining controllerless remote duration. NSG will establish DR Seed based outbound security
associations (SAs), and delete the SAs based on the regular Seed. Upon expiration of the timer, all IPsec policies and
established security associations to the remote NSGs are deleted.

Note: Remote NSGs (that may or may not be controllerless) will also receive updates to transition to the DR Seed
for outbound SAs to the controllerless NSG. Thus, while the sending and receiving NSGs transition to using the DR
Seed there may be some service impact.

If, at any point, connectivity to the VSC is restored and the seed material is received, then the NSG will establish
outbound security associations using the regular Seed. Corresponding remote NSGs will also receive updates to
transition out of DR mode.

3.3. Gateway Operations 104

 VNS User Guide, Release 5.3.2

Alarms for Gateway Operational States

For a given NSG instance, the NSG operational states may be viewed by clicking on the alarm icon in the right
panel. An alarm detail shows one of the following states:
• Controller-less gateway: Indicates that the NSG is in controller-less mode. As explained earlier, the NSG enters
this mode 180 seconds after losing controller connectivity and it may take up to a minute for the state to be
reflected on the VSD.
• Gateway is disconected: Indicates that the NSG is disconnected from the VSC and requires further investigation.
The state itself is only suggestive of connectivity issues and does not provide any visibility into the actual
operational state of the NSG (e.g. it may be within 180 seconds of being disconnected from controller, or be
completely deactivated or operating within the deactivation timer if enabled).

Note: It may take up to a minute for the state changes to be reflected on the VSD.

3.3.8 On-Demand Gateway Deactivation

This section describes the procedure to manually deactivate (revoke) an NSG from the VSD UI.
After a gateway has been bootstrapped, if the gateway is suspected of being compromised in any way, then for se-
curity reasons, an administrator can revoke the NSG certificates via the VSD interface, and reset the gateway to
pre-bootstrapped stage. On revoking NSG certificates from VSD, VSD notifies the VSC that gateway configuration
has changed. VSC via “JSON RPC” notifies NSG that the certificates have been revoked. Upon receiving the notifi-
cation, NSG revokes the certificates and reboots, and resets to pre-bootstrap stage. In case NSG-VSC-VSD is down,
NSG will have to wait for synchronization window when it will re-fetch the infrastructure configuration to learn via
proxy that it has been deactivated.
For manual deactivation the administrator will use the following workflow:
1. Select a Network Service Gateway instance

2. Select the Bootstrap configuration option on the right .

3. Select the Deactivate button to revoke the certificates.

3.3. Gateway Operations 105

 VNS User Guide, Release 5.3.2

Fig. 3.65: Revoke NSG Certificates

On-Demand Gateway Deactivation via CLI

A bootstrapped NSG can also be deactivated via CLI. See CLI-Based NSG Operations (page 106) for more informa-
tion.

3.3.9 CLI-Based NSG Operations

Some NSG operations which are usually executed from the VSD UI can also be executed via CLI. CLI-based NSG
operations can be useful if a VSD UI is not available, or if certain VSD requirements need to be bypassed.

Note: The nuage-operation command must be executed using sudo.

The following options can be used as part of the nuage-operation command:

• [-h], [–help] - This option displays the help message and supported options.
• [-o REQUEST], [–operation REQUEST] - This option specifies whether a REVOKE, REGENERATE, UP-
GRADE or EXPORT_CORE_DUMPS operation is to be executed.
• [–force] - This option forces the operation to be executed without a confirmation prompt.
• [–listpatch] - This option displays a list of patches applied to the NSG.
• [–version] - This option shows the program’s version number and exits.
• [-f FILE], [–file FILE] - This option specifies the NSG image TAR file for pre-bootstrap upgrades.
• [–clean] - This option removes the upgrade files following a successful upgrade.
• [–ignoremd5] - This option tells the NSG to ignore the MD5 file requirement during upgrades. Use at your own
risk.

3.3. Gateway Operations 106

 VNS User Guide, Release 5.3.2

Note: If the nuage-operation command is executed from VSC tools, the --force option is required.

The following sample shows the output for the --help option:

-bash-4.2$ sudo nuage-operation --help

usage: nuage-operation [-h] [-o REQUEST] [--force] [--listpatch] [--version]
[-f FILE] [--clean] [--ignoremd5]

Executes manually triggered commands and operations on the NSG.

optional arguments:
-h, --help show this help message and exit

-o REQUEST, --operation REQUEST

the operation requested for execution. Available :
REVOKE, REGENERATE, UPGRADE and EXPORT_CORE_DUMPS
--force forces the operation, bypassing the confirmation request.
--listpatch will display list of patches applied on this NSG.
--version show program's version number and exit
-f FILE, --file FILE NSG image TAR file
--clean when used with file option, this will clear the
downloaded files post upgrade.
--ignoremd5 will ignore the md5 file for upgrade purposes.

Special operations that may be executed on the device, by user nuage, that are
˓→related to the life cycle of the NSG. It is important to note that operations

˓→triggered by this executable may cause, or is expected to cause, service

˓→interruption.

This utility must be run with sudo.

NSG image upgrade (pre-bootstrap)

Performing a pre-bootstrap NSG image upgrade allows pre-staging of an NSG running a factory-provided gold image
without connecting to a VSD. As the Nuage user, execute the sudo nuage-operation -o UPGRADE command
to perform an upgrade from a USB drive containing the image .tar file and MD5 checksum.
To perform an upgrade using a locally available image .tar file, execute sudo nuage-operation -o UPGRADE
-f <filename> or sudo nuage-operation -o UPGRADE --file <filename> to perform an up-
grade using a locally available .tar file. The MD5 checksum must be in the same directory as the .tar file.
Adding the --clean option to the upgrade command deletes the .tar and image files following a successful upgrade.

Note: The --ignoremd5 allows you to perform the upgrade without validating the MD5 checksum file, or without
the checksum file existing. Use this option at your own risk, as an invalid image file can render the NSG inoperable.

See Pre-bootstrap NSG image upgrades (page 72) for more information about using this command.

Gateway Deactivation

A bootstrapped NSG can be deactivated via CLI. CLI-based deactivation can be useful if the NSG has lost connectivity
to the VSD and if someone cannot be sent to the site to reset the device.

3.3. Gateway Operations 107

 VNS User Guide, Release 5.3.2

Log in to the NSG via SSH or console as the nuage user, and execute the sudo nuage-operation -o REVOKE
command to initiate gateway deactivation. A confirmation prompt warns the user that this command may be service-
impacting.
When the deactivation command is executed, the NSG attempts to communicate the deactivation to the VSD three
times. If communication fails, the NSG proceeds with a local deactivation regardless. The VSD operator must also
make sure the NSG instance is revoked from the VSD before the NSG can be bootstrapped again.

Certificate Renewal

NSGs uses VSPCA signed certificate that is valid for two years. The certificate is renewed automatically when the NSG
is upgraded. Certificate can be renewed without an NSG upgrade by executing the CLI-based certificate regeneration
command. The NSG must have connectivity to the VSD in order for the certificate renewal to be successful.
Log in to the NSG as the nuage user, and execute the sudo nuage-operation -o REGENERATE command to
initiate certificate renewal. A confirmation prompt warns the user that this command may be service-impacting.

Note: Renewing NSG certificates may be service-impacting. Services using TLS, such as OVS and the statistics
collector, should be restarted to ensure they refer to the new certificate. To ensure these services are restarted, reboot
the NSG after the certificate is renewed.

NSG Patch List

The NSG patch list displays a list of all patches successfully applied to the NSG. Log in to the NSG as the nuage user,
and execute the sudo nuage-operation patch-list command to view the patch list.
See NSG Patch Profiles (page 114) for more information about NSG patch operations.

Export core dumps

The Nuage user does not have access to the directory where core dumps are stored following an NSG crash. The sudo
nuage-operation -o EXPORT_CORE_DUMPS command grants the Nuage user read access to /home/abrt/,
which contains the core dump files. After running the command, the Nuage user can view the files for copy them to
another location (using SCP, for example).

3.3.10 Setting up Secure Access to NSG

SSH is used to provide secure remote access to NSGs deployed in branch locations. The following are key capabilities
of the feature:
• Ability to enable/disable SSH service
• Password-based and/or key-based authentication
• Root login is disabled, and instead user nuage is enabled by default and the username is configurable. The
non-root user belongs to a special user group nuage.
• SSH connection uses a non-standard port 893 instead of the standard port 22.
• Authorized user is allowed to run a limited set of white-listed commands.
When an NSG is bootstrapped, a default username and password is set for password-based authentication mode. The
CSPRoot is expected to change the default settings as part of the NSG configuration process using the following
CSPRoot workflow:

3.3. Gateway Operations 108

 VNS User Guide, Release 5.3.2

1. Create Infrastructure Access Profile (page 27) to:

• Optionally configure a username (default is nuage).
• Select the authentication mode to be password-based, key-based, or both. For key-based, SSH public keys will
also have to be added.
• Change the default password by configuring a new password. The same password is also used for console
access, and hence is required even if password-based authentication is disabled for SSH.
• Enable or disable source IP filter which limits the hosts that can connect to the NSG. When enabled, a list of IP
addresses has to be configured.
2. Assign the Infrastructure Access Profile to a NSG Template (page 38).
3. Enable/Disable SSH at the template level.
4. Specify whether an enterprise administrator can override the settings at the instance level.

Note: If a NSG Template is updated such that an Infrastructure Access Profile is disassociated, and if no other
Infrastructure Profile is associated, then the password from the last associated profile will be used.

The enterprise administrator workflow relevant to SSH provisioning is as follows:

1. Instantiate the NSG (page 51) which will inherit all the configuration of the associated template. However,
if the CSPRoot has allowed override of SSH attributes, then as part of the instance creation, the enterprise
administrator may change the SSH policy.
2. The SSH configuration setting is displayed in the System Information panel on the far right, specifically:
• SSH Service Instance displays the configuration at the NSG instance level. Valid values are enabled, disabled
or inherited. If the value displayed is Inherited, then the template level SSH policy applies. Otherwise the
instance level policy takes precedence.
• Inherited SSH Service is the template level SSH policy set by the CSPRoot. Valid values are Enabled or
Disabled.

3.3. Gateway Operations 109

 VNS User Guide, Release 5.3.2

Fig. 3.66: SSH Configuration Information

Note: If CSPRoot disables SSH in the template but allows override, then the enterprise administrator may enable it at
the instance level. In such a scenario, the SSH Service Instance will display Enabled and the Inherited SSH Service
will display Disabled. However, the instance level policy will apply.

The white-listed commands on the NSG are:

ovs-vsctl
ovs-appctl
ovs-appctl bridge/dump-conntracks-summary alubr0
ovs-appctl bridge/dump-conntracks alubr0

3.3. Gateway Operations 110

 VNS User Guide, Release 5.3.2

ovs-ofctl
ovs-dpctl
conntrack -L
iptables -L
iptables -L -n
iptables -t nat -L
iptables -t nat -L -n
dmsetup status
ip link set * up
ip link set * down
ntpq -p -n
ntpq -p
shutdown
poweroff
reboot
nuage-qos-config
conntrack
tcpdump
nuage-vlan-config
vrs-rgctl
ping
traceroute
nslookup
dig
nuage-operation
nuage-flow-top

The following are VSAP wrapper commands:

configure
show
debug

3.3.11 Trusted Platform Module (TPM) Status

TPM capability was introduced in VNS Release 3.2.R4, and was enabled and applied transparently when NSGs were
bootstrapped. However, there was no explicit mechanism to verify TPM support and enablement. In order to provide
more visibility into the hardware capability, the feature is being introduced in Release 4.0.R1 to report the TPM status
(to the VSD) during NSG bootstrapping.
TPM chip has a unique and secret RSA key burned in as it is produced, and in VNS is being used for encrypting the
private key that is generated as part of the NSG certificate process during bootstrapping. The encrypted blob is stored
on the compact flash file system and can only be decrypted using the TPM private key. Thus, TPM enables binding
of the storage (e.g. compact flash) to the motherboard. If the storage is moved to another appliance, and if there is a
TPM mismatch, then it will not be possible to decrypt the private key thus ensuring security and integrity of the NSG
software and certificate (used for authenticating the NSG). It will also isolate the NSG and not allow it to connect to
the VSC/VSD.

Note: The feature only applies to Nuage provided NSG appliances, such as NSG-E and other physical form factors
on the roadmap.

Specifically, the status will indicate one of the following states:

3.3. Gateway Operations 111

 VNS User Guide, Release 5.3.2

• Enabled and operational: When NSG agent is able to successfully detect and load the TPM drivers, run a
self-test, and generate the key which is encrypted using TPM before storing on storage device.
• Disabled: If TPM is disabled in BIOS.
• Unknown: This status is shown in situations, such as:
– NSG-V instances which have no TPM, or 3rd party platforms.
– NSG instance is created in VSD, but it is not actually bootstrapped.
– If VSD is upgraded to Release 4.0.R1 but the NSGs are running a previous version when there was no
status reporting.
• Enabled and not operational: This will probably never be seen as a status. In this scenario, the TPM is detected
as being enabled in BIOS, but for some reason the drivers are not loaded.
To view the status, the CSPRoot or Enterprise administrator can navigate to Infrastructure –> Network Service Gate-
ways –> NSG instance –> and the TPM Status is displayed under the Device Information.

Fig. 3.67: Example TPM Status

The TPM status is reported during bootstrapping once the NSG has successfully authenticated itself to the VSD or has
successfully renewed the certificate. If TPM is disabled or in unknown state, NSG will continue with the bootstrap.
However, when a TPM-enabled deactivated NSG is being re-bootstrapped, if key generation fails, then bootstrapping
is aborted.
During upgrade from 3.2 to 4.0 environments, the TPM status is reported as follows:
1. VSD Upgrade
When the VSD is upgraded from 3.2 to 4.0, any discovered and managed NSGs that were in the 3.2 database, will be
upgraded to 4.0 with a TPM Status value set to Unknown. That is because even though the NSG was using TPM, the
VSD in 3.2 was not able to track the TPM status of NSGs bootstrapped at that time.
2. NSG Upgrade
When the NSG is being upgraded, there are two scenarios:
• NSG is running version 3.2 R5+ and is TPM enabled. Since TPM is already in use, the NSG will authenticate
successfully and will not renew the certificates. As a consequence, there will not be an opportunity to report the
correct TPM status and status will remain as Unknown.
• NSG is using the 3.2 Gold image when TPM was not supported. When the upgrade to 4.0 is performed and the
NSG reboots, it will detect that TPM is not being used despite being enabled in the BiOS (will not apply for
VMs). Since TPM is supported in 4.0, the NSG will renew the certificate and report the status as Enabled and
Operational.

3.3. Gateway Operations 112

 VNS User Guide, Release 5.3.2

For any other upgrade cases, not covered by the above, a status of unknown will be reported.

3.3.12 NSG Upgrade Profiles

NSG upgrade profiles can be used to upgrade a single NSG instance without performing a reload configuration or
modifying an NSG template. This allows you to perform an upgrade on an NSG without impacting services provided
by other NSGs in the network.

Note: For NSGs release 5.1.1 and later, NSG upgrade profiles must be used to upgrade the NSG. Earlier releases of
the NSG do not support NSG upgrade profiles.

NSG Upgrade Profiles Workflow

Perform the following steps to upgrade an NSG using an NSG upgrade profile. This workflow assumes that the NSG
is bootstrapped and operational.
Create an NSG upgrade profile

1. As CSProot, click on the Open Platform Configuration icon and select the Infrastructure tab.

2. Click on the Network Services Gateways , NSG Profiles , and Upgrade Profiles buttons.

3. Click on the button to create a new NSG upgrade profile.

4. Configure the Name, Description, and Upgrade Metadata URL parameters
The Upgrade Metadata URL parameter defines the path to the upgrade meta-
data file. You must ensure that a fully qualified path is entered (e.g.,
https://<server:port>/<path>/upgrade_metadata.json). Any port number may be
used, as long as the same number port is opened on the proxy to redirect upgrade requests from NSG to the web
server.
5. Click Create.
Assign the profile to an NSG

6. Click on the Network Services Gateways button.

7. Right-click an NSG and choose Edit.
8. Choose the created NSG upgrade profile for the Upgrade Profile parameter and click Update.
Execute the NSG upgrade
9. Click on the Upgrade Management button at the bottom of the NSG list panel.
10. Click Download and Upgrade to execute the upgrade.

3.3. Gateway Operations 113

 VNS User Guide, Release 5.3.2

Note: If you click Download, the NSG software image is downloaded, but the NSG is not upgraded.

11. Click on the Command button in the right-hand information panel for the selected NSG.
12. Observe the upgrade progress.
When the upgrade is complete, the NSG is automatically rebooted. The Command panel includes status infor-
mation detailing the upgrade.

3.3.13 NSG Patching

The NSG patch framework is introduced in 5.3.1 to support BIOS and TPM firmware upgrades for activated physical
NSGs. This section describes how to use the patch framework to apply such patches on an NSG instance or a group
of NSGs. For more information on the NSG patches available please contact the Nuage Networks support.

NSG Patch Profile

A patch profile defines the URL to a patch bundle which must be made available on a HTTP or HTTPS server
accessible by the NSG. When the csproot user applies the profile on the NSG instance and initiates the patching

3.3. Gateway Operations 114

 VNS User Guide, Release 5.3.2

command, the NSG downloads the patch and applies it. The status of the patch can be monitored from the Operations
panel in the VSD.

NSG Patch Bundle

The NSG patch bundle is an RPM bundle containing the files required to patch an NSG. The RPM filename follows
the following format:
nuage-nsg-patch-<name>-<version>-<release>.x86_64.rpm
For example:
nuage-nsg-patch-tpm-upgrade-1.0-0.x86_64.rpm
The name, version, and release values in this format are identified by the VSD as the patchtag. In the above example,
the patchtag is tpm-upgrade-1.0-0. The patchtag is a unique identifier used by the VSD. The RPM file should
not be renamed, as this could result in verification errors. The RPM filename format is case sensitive, so although
capital letters are allowed, they should be avoided.
The patch.info file in the patch bundle is a JSON file that contains the unique patchtag information verified on the
NSG. It also includes the patch name, version, release, and description.
When the patch bundle is downloaded, it should be hosted on a web server reachable by the NSG. This server is
typically the same server used to host software images during an NSG upgrade. The server must have a VSPCA
signed certificate and may also require TLS client certificate verification. HTTP or HTTPS web servers are supported,
but HTTPS is highly recommended.

Note: The patch download and implementation may take longer depending on the size and functionality of the patch.
For example, a TPM patch may take about 20 minutes to be applied. This may result in the NSG being disconnected
for longer than the auto-deactivation timer. Refer to the release notes provided with the patch for more information
about the estimated running time for the patch implementation. The release notes may also state whether the auto-
deactivation timer should be disabled or increased before beginning the patch download.

Detailed Patch Monitoring

An NSG patch operation can be monitored via status messages on the VSD. In addition to this, the patch status and log
files can be viewed during a patch operation. These files provide more detailed historical event logging for the patch
operation.
• The cat patch.info (cd /home/root/patches/<patch_tag>) command displays patch information. This may
include the expected running time for patch implementation, which could indicate whether a patch operation is
taking longer than expected.
• The cat patch_status.json (cd /home/root/patches/) command displays status messages for the patch
operation.
• The tailf patch.log (cd /home/root/patches/<patch_tag>) command displays log details for the patch
operation.
Refer to Patch Operation Status Codes (page 117) for a list of all status messages.
Refer to this sample output (page 127) for an example of a failed status message.

3.3. Gateway Operations 115

 VNS User Guide, Release 5.3.2

Patch Status Scenarios

The cat patch_status.json command displays a patch operation status along with a status code. When a
patch is successfully downloaded and applied, the following statuses and codes are displayed:
1. Status: “STARTED”; Status code: 0 (Unknown)
2. Status: “RUNNING”; Status code: 13 (Request Received)
3. Status: “RUNNING”; Status code: 110 (Patch Installation Started)
4. Status: “COMPLETED”; Status code: 17 (Command Complete)
When a TPM patch operation is attempted while another patch is being applied to the NSG, the following statuses and
codes are displayed:
1. Status: “FAILED”; Status code: 115 (TPM already at patch version)
2. Status: “ABANDONED”; Status code: 0 (Unknown)

NSG Patch List

The NSG patch list displays a list of all successfully applied patches. Execute sudo nuage-operation
patch-list to view the NSG patch list.

NSG Patch Profiles Workflow

Perform the following steps to patch an NSG using an NSG patch profile. This workflow assumes that the NSG is
bootstrapped and operational.
Create an NSG patch profile

1. As CSProot, click on the Open Platform Configuration icon and select the Infrastructure tab.

2. Click on the Network Services Gateways , NSG Profiles , and Patch Profiles buttons.

3. Click on the button to create a new NSG patch profile.

4. Configure the Name, Description, and Patch URL parameters.
The Patch URL parameter defines the path to the NSG patch. You must ensure that a fully qualified path is
entered (e.g., https://downloads.com/nuage-nsg-patch-tpm-upgrade-1.0-0.rpm).

Note: Both http and https patch bundle URLs are supported.

5. Click Create.
Assign the profile to an NSG

6. Click on the Network Services Gateways button.

7. Click on the Upgrade Management button at the bottom of the NSG list panel. The NSG Download Upgrade
and Patch Management window opens.
8. Select the NSG Patch tab.
9. Choose the created NSG patch profile for the NSG Patch Profile parameter.

3.3. Gateway Operations 116

 VNS User Guide, Release 5.3.2

10. Click Apply NSG Patch.

If the NSG is bootstrapped and the patch URL is valid, the NSG begins downloading and applying the patch.

Note: Some patches may require the NSG to be rebooted once the patch is applied.

Note: If a patch is already being applied to the selected NSG, a server message appears and states
that the action cannot be performed. The patch operation fails with error code - “unknown”.

Monitor the patch status

11. Click on the Command icon in the NSG info panel.
12. View the Operations panel which lists completed and in-progress operations along with their statuses. The
displayed status will be “STARTED”, “RUNNING”, “FAILED”, “ABANDONED”, or “COMPLETED”.

13. If required, monitor the detailed progress of the patch implementation using the patch info, status, and log files.
• Execute cat patch.info (cd /home/root/patches/<patch_tag>) to view patch information. This may include
the expected running time for patch implementation.
• Execute cat patch_status.json (cd /home/root/patches/) to view status messages for the patch opera-
tion.
• Execute tailf patch.log (cd /home/root/patches/<patch_tag>) to view log details for the patch operation.

Note: The patch download and implementation may take longer depending on the size and functionality of the patch.
Note the estimated running time for the patch operation and allow time for the VSD to complete the operation.

Note: You can abandon a patch operation in progress by clicking the red square button next to the operation. A patch
operation may need to be abandoned if the patch update was interrupted directly on the NSG. In this case, the status
update may not be updated in the VSD.

Note: An NSG upgrade operation cannot be initiated during an NSG patch operation.

3.3.14 Patch Operation Status Codes

This section lists the status codes that can be displayed using cat patch_status.json during an NSG patch
operation.

3.3. Gateway Operations 117

 VNS User Guide, Release 5.3.2

Unknown Status

The unknown status command is returned when the VSD receives a status code it does not recognize.

CODE_00000(0, "UNKNOWN", "Unknown status."),

NSG Upgrade Statuses

CODE_00001(1, "Upgrade started", "The NSG has received an upgrade request.",

˓→OperationStatus.RUNNING),

CODE_00002(2, "Upgrade scheduled", "The NSG has received a request to schedule an

˓→upgrade.", OperationStatus.RUNNING),

CODE_00003(3, "Immediate upgrade", "The NSG is initialising an immediate upgrade.",

˓→OperationStatus.RUNNING),

CODE_00004(4, "Download started", "The NSG has started downloading the upgrade image.
˓→", OperationStatus.RUNNING),

CODE_00005(5, "Download complete", "The NSG has completed the download of the upgrade
˓→image.", OperationStatus.COMPLETED),

CODE_00006(6, "File extraction", "The NSG is extracting files from the downloaded
˓→image.", OperationStatus.RUNNING),

CODE_00007(7, "Upgrade complete", "The NSG is now rebooting to complete upgrade.",

˓→OperationStatus.COMPLETED),

CODE_00008(8, "Server unreachable", "The NSG is unable to reach the upgrade server.",
˓→OperationStatus.FAILED),

CODE_00009(9, "No file", "The NSG has not found any file to download.",
˓→OperationStatus.FAILED),

CODE_00010(10, "Already upgraded", "The NSG is already at the latest version.",

˓→OperationStatus.FAILED),

CODE_00011(11, "Image corruption", "The downloaded image file is corrupted.",

˓→OperationStatus.FAILED),

CODE_00012(12, "Disk space error", "The NSG is out of disk space to proceed with an
˓→upgrade.", OperationStatus.FAILED),

CODE_00013(13, "Request received", "The NSG has received a command execution request.
˓→", OperationStatus.RUNNING),

CODE_00014(14, "Command executing", "The requested operation is being executed.",

˓→OperationStatus.RUNNING),

CODE_00015(15, "Invalid data", "No data or invalid data was received with command
˓→request.", OperationStatus.FAILED),

CODE_00016(16, "Incomplete data", "Not enough information was received to accomplish

˓→the operation.", OperationStatus.FAILED),

CODE_00017(17, "Command complete", "The execution of the command returned

˓→successfully.", OperationStatus.COMPLETED),

CODE_00018(18, "File permission", "Permissions to the upgrade directory on the NSG

˓→are too restrictive.", OperationStatus.FAILED),

CODE_00019(19, "Installation failed", "The installation of the upgrade image failed.

˓→Extraction or saving failure.", OperationStatus.FAILED),

CODE_00020(20, "Execution conflict", "The requested command can't be executed,

˓→another one is already running.", OperationStatus.FAILED),

CODE_00021(21, "Unexpected error", "An unexpected error was detected while performing
˓→the operation.", OperationStatus.FAILED),

NSG Patch Statuses

3.3. Gateway Operations 118

 VNS User Guide, Release 5.3.2

CODE_00100(100, "Unknown Patch Error", "An unknown patch error has been reported.
˓→Check NSG logs for more details.", OperationStatus.FAILED),

CODE_00101(101, "RPM Download Error", "Error during download of RPM.",

˓→OperationStatus.FAILED),

CODE_00102(102, "Failed to acquire lock", "Could not acquire lock in order to start
˓→the RPM installation.", OperationStatus.FAILED),

CODE_00103(103, "Script execution error", "Execution of script verify.py has reported

˓→an error.", OperationStatus.FAILED),

CODE_00104(104, "Script execution error", "Execution of script install.py has

˓→reported an error.", OperationStatus.FAILED),

CODE_00105(105, "RPM install failed", "Installation of RPM failed.", OperationStatus.

˓→FAILED),

CODE_00106(106, "RPM signature verification failed", "Signature verification of RPM

˓→failed.", OperationStatus.FAILED),

CODE_00107(107, "Patch already installed", "This patch was already successfully

˓→installed.", OperationStatus.FAILED),

CODE_00108(108, "Patch tag does not match with RPM", "The patch tag sent from VSD was
˓→not the same as the RPM.", OperationStatus.FAILED),

CODE_00109(109, "Unable to load patch info file", "The patch info file from the RPM
˓→could not be loaded.", OperationStatus.FAILED),

CODE_00110(110, "Patch Installation Started", "NSG has triggered install procedure

˓→for the patch.", OperationStatus.RUNNING),

CODE_00111(111, "Command status update failed", "Failed to update the status of

˓→command in VSD.", OperationStatus.FAILED),

CODE_00112(112, "Public Key not found", "Nuage GPG Public Key was not found.",
˓→OperationStatus.FAILED),

CODE_00113(113, "Script execution error", "Patch resuming script has returned with an
˓→error.", OperationStatus.FAILED),

CODE_00114(114, "Script execution error", "Execution of the post-processing

˓→verification script has reported an error.", OperationStatus.FAILED),

CODE_00115(115, "TPM already at patch version",

"The NSG is reporting that its TPM version is already at the same version as the
˓→one in the patch", OperationStatus.FAILED),

CODE_00116(116, "No TPM detected on device",

"The NSG has reported that there is no visible TPM to patch, verify that this is
˓→not a NSG-V.", OperationStatus.FAILED),

CODE_00117(117, "Invalid device for patch",

"The patch is being installed on a NSG that doesn't support it, verify the chassis
˓→type.", OperationStatus.FAILED),

CODE_00118(118, "BiOS preventing patching", "The NSG BiOS is incompatible with the
˓→patch being applied.", OperationStatus.FAILED),

CODE_00119(119, "Patch Error", "Check NSG logs for more details.", OperationStatus.
˓→FAILED),

CODE_00120(120, "Patch Error", "Check NSG logs for more details.", OperationStatus.
˓→FAILED),

3.3.15 Datapath CLI commands

The following commands provide information about active flows on the NSG.

nuage-flow-top

This command provides an interactive summary output of various kernel flow metrics. The command automatically
updates every one (1) second. To exit the interactive command, press Ctrl+C. The following is a sample output:

3.3. Gateway Operations 119

 VNS User Guide, Release 5.3.2

Flows added to kernel/sec: 7

Flows deleted from kernel/sec: 6
Current flows present in kernel: 28
Average flows present in kernel: 5

The displayed fields are the following:

• Flows added to the kernel/sec: Total number of unique flows that have been successfully installed in the OVS
kernel table in the last second.
• Flows deleted from kernel/sec: Total number of unique flows that have been successfully deleted from the OVS
kernel table in the last second.
• Current flows present in kernel: Total number of flows present in the OVS kernel table.
• Average flows present in kernel: Average number of flows present in the kernel table.

ovs-appctl bridge/dump-conntracks-summary alubr0

This command outputs a summary of the total connections that OVS is tracking in order to maintain state. OVS
will track connection state whenever any of the following features are used: stateful ACLs, Application Discovery,
Application Performance Monitoring (APM), L7 ACLs, PAT/NAT in the overlay.
-bash-4.2$ ovs-appctl bridge/dump-conntracks-summary alubr0
Total | Connecting | Established | Closing | Dpi_in_progress | Dpi_done
-------+------------+-------------+--------+---------------+---------
37 | 13 | 22 | 2 | 10 | 10
SYN-sent | SYN-recvd | SYN-sent-recvd
---------+-----------+----------------
11 | 20 | 0

L7_classification miss: 0
Max supported Connections: 250000

ovs-appctl bridge/dump-conntracks alubr0

This command outputs the total list of connections that OVS is tracking in order to maintain state. OVS will track
connection state whenever any of the following features are used: stateful ACLs, Application Discovery, Application
Performance Monitoring (APM), L7 ACLs, PAT/NAT in the overlay.
-bash-4.2$ ovs-appctl bridge/dump-conntracks alubr0
port | proto | nw_src |
˓→ nw_dst | tp_src | tp_dst | timeout | expires | state |
˓→ detail | user | connId
------+-------+-----------------+-----------------+--------+--------+---------+-------
˓→--+-----------------+-----------------+------+--------------

8 | 6 | 10.10.10.22 |
˓→ 10.31.140.25 | 50350 | 8443 | 180 | 176 | CONN_STATE_INIT |
˓→ DPI_DONE | mirror | 139831386063456
8 | 17 | 10.10.10.22 |
˓→ 125.2.5.145 | 60746 | 53 | 180 | 163 | CONN_STATE_INIT |
˓→ DPI_DONE | mirror | 139831385388192
8 | 6 | 10.10.10.22 |
˓→ 216.58.193.163 | 50698 | 443 | 180 | 77 | CONN_STATE_INIT |
˓→ DPI_DONE | mirror | 139831384599184
8 | 6 | 10.10.10.22 |
˓→ 40.97.155.178 | 50725 | 443 | 180 | 80 | CONN_STATE_INIT |
˓→ DPI_DONE | mirror | 139831384814016

3.3. Gateway Operations 120

 VNS User Guide, Release 5.3.2

8 | 6 | 10.10.10.22 |
˓→ 151.101.56.102 | 50760 | 443 | 180 | 158 | CONN_STATE_INIT |
˓→ DPI_DONE | mirror | 139831384825408
8 | 6 | 10.10.10.22 |
˓→ 216.58.193.161 | 50661 | 443 | 180 | 73 | CONN_STATE_INIT |
˓→ DPI_DONE | mirror | 139831385500048
8 | 6 | 10.10.10.22 |
˓→ 104.244.42.129 | 50260 | 443 | 180 | 160 | CONN_STATE_INIT |
˓→ DPI_DONE | mirror | 139831385583392
8 | 6 | 10.10.10.22 |
˓→ 10.10.10.1 | 50757 | 8013 | 180 | 3 | CONN_STATE_INIT |
˓→ DPI_DONE | mirror | 139831385346752
8 | 6 | 10.10.10.22 |
˓→ 216.58.193.163 | 50655 | 443 | 180 | 76 | CONN_STATE_INIT |
˓→ DPI_DONE | mirror | 139831385294416
8 | 6 | 10.10.10.22 |
˓→ 151.101.56.102 | 50761 | 443 | 180 | 142 | CONN_STATE_INIT |
˓→ DPI_DONE | mirror | 139831385381424
8 | 6 | 10.10.10.22 |
˓→ 64.233.185.189 | 49775 | 443 | 180 | 177 | CONN_STATE_INIT |
˓→ DPI_DONE | mirror | 139831385349408

The displayed columns are the following:

• port: internal port being used on the OVS bridge.
• proto: L4 Protocol in use (6 TCP, 17 UDP).
• nw_src: Source IP of the connection.
• nw_dst: Destination IP of the connection.
• tp_src: Source port number.
• tp_dst: Destination port number.
• timeout: Timeout in seconds for the connection.
• expires: Time in seconds before the connection expires.
• state: Connection state.
• detail: State of the DPI process for that connection.
• user: Internal datapath destination.
• connID: Unique identifier for the connection.

conntrack -L

This command lists all of the connections that conntrack is tracking in the kernel, as well as their state. Connections
will be tracked in the kernel when PAT/NAT functions to the underlay are in use. For further information, refer to the
Linux conntrack manual (man conntrack).
-bash-4.2$ conntrack -L
tcp 6 431991 ESTABLISHED src=10.10.10.22 dst=40.97.29.226 sport=52957 dport=443
˓→src=40.97.29.226 dst=139.237.211.54 sport=443 dport=52957 [ASSURED] mark=532738

˓→use=1

udp 17 20 src=10.10.10.22 dst=125.2.5.145 sport=54831 dport=53 src=125.2.5.145

˓→dst=139.237.211.54 sport=53 dport=54831 mark=532738 use=1

tcp 6 431983 ESTABLISHED src=10.10.10.22 dst=64.233.185.189 sport=53018

˓→dport=443 src=64.233.185.189 dst=139.237.211.54 sport=443 dport=53018 [ASSURED]

˓→mark=532738 use=1

3.3. Gateway Operations 121

 VNS User Guide, Release 5.3.2

udp 17 17 src=10.10.10.22 dst=125.2.5.145 sport=63350 dport=53 src=125.2.5.145

˓→dst=139.237.211.54 sport=53 dport=63350 mark=532738 use=1

tcp 6 431998 ESTABLISHED src=135.227.222.54 dst=124.252.253.122 sport=53767

˓→dport=7407 src=124.252.253.122 dst=139.237.211.54 sport=7407 dport=53767 [ASSURED]

˓→mark=0 use=1

tcp 6 431991 ESTABLISHED src=10.10.10.22 dst=40.97.169.146 sport=53513 dport=443

˓→src=40.97.169.146 dst=139.237.211.54 sport=443 dport=53513 [ASSURED] mark=532738

˓→use=1

...output omitted...
conntrack v1.4.2 (conntrack-tools): 55 flow entries have been shown.

3.3.16 CLI Output

Whitelisted NSG commands can be used to check the configuration for controllerless operations.
In the command output below, the line controller_less_mode: remote shows the mode.

# ovs-appctl ofproto/show alubr0

Name:alubr0 version: (null)
dp_id: 0x20f96793 gen_id: 0x1 cleanup_pending:0
local_flows_cleanup_pending: no remote_flows_cleanup_pending:
˓→ no
personality: nsg vrs_role: master
acl_tcp_timeout: 3600(sec) acl_non_tcp_timeout: 180(sec)
vport_init_stateful_timer: 300(sec)
mgmt: 0.0.0.0
dhcp-relay: state: Disabled dhcp-sock: -1
controllers: fail_mode: secure
S: ssl:10.10.13.7:6633 state: ACTIVE role: master name: ctrl1
˓→local_ip: 10.15.1.254

S: band: out probe:5 vsc-state:

˓→FUNCTIONAL last-role-change:2016-10-07 23:20:20.272 uplink:port1
S: ssl:10.10.15.9:6633 state: BACKOFF role: slave name:
˓→ctrl2 local_ip: 0.0.0.0
S: band: out probe:5 vsc-state:
˓→UNKNOWN last-role-change:2016-10-07 23:20:18.875 uplink:port1
stats_collectors:
revert-behavior: no
controller_less_mode: remote
datapath_entries:
UUID: a3b0ceb6-074d-4aef-a71b-1e8ea2d8c5c7 cpe_id: 0x0

In the output for the command ovs-appctl ofproto/show alubr0 below, the lines that display the mode and
timer settings are:

controller_less_mode: remote
controller_less_duration: 7 day 0 hr 0 min
controller_less_duration_left: 0 day 13 hr 59 min 58 sec
head _less_duration: 0 day 14 hr 0 min
head_less_duration_left: 0 day 13 hr 59 min 58 sec

# ovs-appctl ofproto/show alubr0

Name:alubr0 version: (null)
dp_id: 0x20f96793 gen_id: 0x2 cleanup_pending:1
local_flows_cleanup_pending: no remote_flows_cleanup_
˓→ pending: no

3.3. Gateway Operations 122

 VNS User Guide, Release 5.3.2

personality: nsg vrs_role: master

acl_tcp_timeout: 3600(sec) acl_non_tcp_timeout: 180(sec)
vport_init_stateful_timer: 300(sec)
mgmt: 0.0.0.0
dhcp-relay: state: Disabled dhcp-sock: -1
controllers: fail_mode: secure
S: ssl:10.10.13.7:6633 state: CONNECTING role: slave name: ctrl1
˓→ local_ip: 10.15.1.254
S: band: out probe:5 vsc-state:
˓→UNKNOWN last-role-change:2016-10-08 00:21:20.276 uplink:port1
S: ssl:10.10.15.9:6633 state: BACKOFF role: slave name:
˓→ctrl2 local_ip: 0.0.0.0
S: band: out probe:5 vsc-state:
˓→UNKNOWN last-role-change:2016-10-07 23:20:18.875 uplink:port1
stats_collectors:
revert-behavior: no
controller_less_mode: remote
controller_less_duration: 7 day 0 hr 0 min
controller_less_duration_left: 0 day 13 hr 59 min 58 sec
head_less_duration: 0 day 14 hr 0 min
head_less_duration_left: 0 day 13 hr 59 min 58 sec
datapath_entries:
UUID: a3b0ceb6-074d-4aef-a71b-1e8ea2d8c5c7 cpe_id: 0x0

The output for the following command displays the Local DR Mode, Remote DR Mode, Customer ID and VRFs.
# ovs-appctl ipsec/list-policies
IPSec policies:
Local DR Mode: Yes
VxLAN src VxLAN dst ESP local ESP remote UDP dport
˓→Remote-DR Customer-ID VRFs
--------------------------------------------------------------------------------------
˓→---------------------------

32.249.103.147 46.102.252.247 10.15.1.254 10.15.3.254 4500

˓→No 10004 (41b8ca8f, 11b841fb)

32.249.103.147 214.78.125.68 10.15.1.254 10.15.33.254 4500

˓→No 10004 (41b8ca8f, 11b841fb)

The following is a new command introduced in 4.0.R5:

# ovs-appctl ipsec/list-dr-seed
Seed entries:
ID Create Time Start Time Time left(sec) Customer-Id
--------------------------------------------------------------------------------------
1475859359001 2016-10-07 16:55:59 2016-10-08 16:56:03 145951 10004
1475859303185 2016-10-07 16:55:03 2016-10-07 16:56:03 59551 10004

The following is a new command introduced in 4.0.R5:

# ovs-appctl ipsec/list-dr-sa
IPSec SAs:
SPI Seed ID Enc Algo Auth Algo Soft life(sec) Hard
˓→life(sec) Customer-Id
--------------------------------------------------------------------------------------
˓→----------------------

0x1b2bb8af 1475859303185 AES_128_CBC HMAC_SHA1 11 29

˓→ 10004

3.3. Gateway Operations 123

 VNS User Guide, Release 5.3.2

The following is a new command introduced in 4.0.R5:

# ovs-appctl ipsec/list-dr-xfrm-state 46.102.252.247

Src Dst Dir SPI SKB mark Customer-ID
------------------------------------------------------------------------------
10.15.3.254 10.15.1.254 In 0xfdf12c69 0x0 10004
10.15.3.254 10.15.1.254 In 0x5631a236 0x0 10004
10.15.3.254 10.15.1.254 In 0x291b55e9 0x0 10004
10.15.3.254 10.15.1.254 In 0xe32756a8 0x0 10004
10.15.3.254 10.15.1.254 In 0xe8ca2761 0x0 10004
10.15.3.254 10.15.1.254 In 0x24aca2ba 0x0 10004
10.15.3.254 10.15.1.254 In 0x3cf24121 0x0 10004
10.15.3.254 10.15.1.254 In 0xa0b56837 0x0 10004
10.15.1.254 10.15.3.254 Out 0x5262b0dc 0x19000100 10004
10.15.1.254 10.15.3.254 Out 0x46df52b0 0x19000120 10004
10.15.1.254 10.15.3.254 Out 0x4249dda6 0x19000128 10004
10.15.1.254 10.15.3.254 Out 0xe076001b 0x19000148 10004
10.15.1.254 10.15.3.254 Out 0x4e1db489 0x19000188 10004
10.15.1.254 10.15.3.254 Out 0xc9cef06 0x190001b8 10004
10.15.1.254 10.15.3.254 Out 0xc0dec450 0x190001c0 10004
10.15.1.254 10.15.3.254 Out 0x945cff7 0x190001e0 10004

In the output for the command show vswitch-controller vswitches detail below, the following is a
new output:

Graceful Restart Support :Disabled

*A:vsc#show vswitch-controller vswitches detail

===============================================================================
Virtual Switch Table
===============================================================================
vswitch-instance : va-10.15.2.254/1
Personality : NSG
Uptime : 0d 01:32:28 VM Count : 0
Graceful Restart Support : Disabled
Num of hostIf : 0 Num of bridgeIf : 0
Num of multiVMs : 0
Num of Containers : 0
OF version : 1 OF nego. version : 1
OF Conn. port : 6633
Cntrl. role : primary Cntrl. Conn. type : tls
Cntrl. crl lookup : false
Cntrl. Conn. mode : secure
Cntrl. Conn. state : ready
Cntrl. client verification : true
Cntrl. client IP verification : false
Peer IP for resiliency : -
Gateway Hold Time(sec) : 15 Gateway Echo Time(sec) : 5
Gateway Topic : nuage_gateway_id_13.11.94.19
Gateway Retry/Audit Time : 1129
XMPP error code : 0
XMPP error text : (Not Specified)
JSON Conn. State : Up
JSON Sess. Uptime : 0d 01:32:26
Static Peer : False
Datapath-Id : va-13.11.94.19

3.3. Gateway Operations 124

 VNS User Guide, Release 5.3.2

-------------------------------------------------------------------------------
Dual Uplink Table
-------------------------------------------------------------------------------
IP Addr Preference Behind NAT PAT
Uplink
-------------------------------------------------------------------------------
10.15.2.254 primary no no
-------------------------------------------------------------------------------
No. of Dual Uplink Mappings: 1
-------------------------------------------------------------------------------
XMPP Tls Profile : n/a
OF Tls Profile : of-tls-profile
Ovsdb Tls Profile : n/a
Ovsdb Conn Type : none
vswitch-instance : va-10.15.33.254/1
Personality : NSG
Uptime : 0d 01:32:31 VM Count : 0
Graceful Restart Support : Disabled
Num of hostIf : 6 Num of bridgeIf : 8
Num of multiVMs : 0
Num of Containers : 0
OF version : 1 OF nego. version : 1
OF Conn. port : 6633
Cntrl. role : primary Cntrl. Conn. type : tls
Cntrl. crl lookup : false
Cntrl. Conn. mode : secure
Cntrl. Conn. state : ready
Cntrl. client verification : true
Cntrl. client IP verification : false
Peer IP for resiliency : -
Gateway Hold Time(sec) : 15 Gateway Echo Time(sec) : 5
Gateway Topic : nuage_gateway_id_214.78.125.68
Gateway Retry/Audit Time : 977
XMPP error code : 0
XMPP error text : (Not Specified)
JSON Conn. State : Up
JSON Sess. Uptime : 0d 01:32:29
Static Peer : False
Datapath-Id : va-214.78.125.68

-------------------------------------------------------------------------------
Dual Uplink Table
-------------------------------------------------------------------------------
IP Addr Preference Behind NAT PAT
Uplink
-------------------------------------------------------------------------------
10.15.33.254 primary no no
-------------------------------------------------------------------------------
No. of Dual Uplink Mappings: 1
-------------------------------------------------------------------------------
XMPP Tls Profile : n/a
OF Tls Profile : of-tls-profile
Ovsdb Tls Profile : n/a
Ovsdb Conn Type : none
-------------------------------------------------------------------------------
No. virtual switches: 2
===============================================================================

3.3. Gateway Operations 125

 VNS User Guide, Release 5.3.2

In the output for the command show vswitch-controller vswitches detail below, the new lines in the
output are:

Graceful Restart Support : Enabled

GR Time Interval : 0d 14:00:00
GR entered count : 6
GR Last entered : 10/08/2016 00:21:25
GR Exit in : 0d 13:52:48

*A:vsc# show vswitch-controller vswitches detail

===============================================================================
Virtual Switch Table
===============================================================================
vswitch-instance : va-10.15.1.254/1
Personality : NSG
Uptime : 0d 00:00:00 VM Count : 0
Graceful Restart Support : Enabled
GR Time Interval : 0d 14:00:00
GR entered count : 6
GR Last entered : 10/08/2016 00:21:25
GR Exit in : 0d 13:52:48
Num of hostIf : 9 Num of bridgeIf : 8
Num of multiVMs : 0
Num of Containers : 0
OF version : 1 OF nego. version : 1
OF Conn. port : 6633
Cntrl. role : secondary Cntrl. Conn. type : tls
Cntrl. crl lookup : false
Cntrl. Conn. mode : secure
Cntrl. Conn. state : none
Cntrl. client verification : true
Cntrl. client IP verification : false
Peer IP for resiliency : -
Gateway Hold Time(sec) : 15 Gateway Echo Time(sec) : 5
Gateway Topic : nuage_gateway_id_32.249.103.147
Gateway Retry/Audit Time : 956
XMPP error code : 0
XMPP error text : (Not Specified)
JSON Conn. State : Down
JSON Sess. Uptime : 0d 00:00:00
Static Peer : False
Datapath-Id : va-32.249.103.147

-------------------------------------------------------------------------------
Dual Uplink Table
-------------------------------------------------------------------------------
IP Addr Preference Behind NAT PAT
Uplink
-------------------------------------------------------------------------------
10.15.1.254 primary yes(full-cone) no
-------------------------------------------------------------------------------
No. of Dual Uplink Mappings: 1
-------------------------------------------------------------------------------
XMPP Tls Profile : n/a
OF Tls Profile : of-tls-profile
Ovsdb Tls Profile : n/a
Ovsdb Conn Type : none

3.3. Gateway Operations 126

 VNS User Guide, Release 5.3.2

vswitch-instance : va-10.15.3.254/1
Personality : NSG
Uptime : 0d 01:33:35 VM Count : 0
Graceful Restart Support : Disabled
Num of hostIf : 8 Num of bridgeIf : 8
Num of multiVMs : 0
Num of Containers : 0
OF version : 1 OF nego. version : 1
OF Conn. port : 6633
Cntrl. role : primary Cntrl. Conn. type : tls
Cntrl. crl lookup : false
Cntrl. Conn. mode : secure
Cntrl. Conn. state : ready
Cntrl. client verification : true
Cntrl. client IP verification : false
Peer IP for resiliency : -
Gateway Hold Time(sec) : 15 Gateway Echo Time(sec) : 5
Gateway Topic : nuage_gateway_id_46.102.252.247
Gateway Retry/Audit Time : 902
XMPP error code : 0
XMPP error text : (Not Specified)
JSON Conn. State : Up
JSON Sess. Uptime : 0d 01:33:28
Static Peer : False
Datapath-Id : va-46.102.252.247

-------------------------------------------------------------------------------
Dual Uplink Table
-------------------------------------------------------------------------------
IP Addr Preference Behind NAT PAT
Uplink
-------------------------------------------------------------------------------
10.15.3.254 primary yes(full-cone) no
-------------------------------------------------------------------------------
No. of Dual Uplink Mappings: 1
-------------------------------------------------------------------------------
XMPP Tls Profile : n/a
OF Tls Profile : of-tls-profile
Ovsdb Tls Profile : n/a
Ovsdb Conn Type : none
-------------------------------------------------------------------------------
No. virtual switches: 2
===============================================================================

The cat patch_status.json command can be executed to view detailed status messages for the current NSG
patch operation. The following sample displays a failed status message.
vsc# cat patch_status.json
{
"Status": "Failed",
"TotalSteps": 6,
"TimestampLastCompletedStep": "2018-04-11T21:43:43.302837",
"TimestampStartingProcedure": "2018-04-11T21:42:30.518058",
"ErrorMessage": "logger: </home/root/patches/tpmfirmwareupgrade-1.0-1/custom/2_
˓→tpmactivateEnable.sh> STEP 2 : Begin Activate and Enable of TPM.\nlogger: </home/

˓→root/patches/tpmfirmwareupgrade-1.0-1/custom/2_tpmactivateEnable.sh> Current TPM

˓→state : Enabled=No; Activated=No; Owned=No\nlogger: </home/root/patches/

˓→tpmfirmwareupgrade-1.0-1/custom/2_tpmactivateEnable.sh> Found that TPM has no

˓→longer an Owner set. Proceeding with firmware update...\nlogger: </home/root/

˓→patches/tpmfirmwareupgrade-1.0-1/custom/2_tpmactivateEnable.sh> Unsupported BiOS

˓→version 3260G118\n",
3.3. Gateway Operations 127
 VNS User Guide, Release 5.3.2

"LastSuccessfulStep": 1,
"ProcedureName": "tpmfirmwareupgrade-1.0-1",
"ProcedureComplete": 0,
"CommandUUID": "56332fa3-0a55-1f12-bcc2-bbd6c61185b5"

3.3. Gateway Operations 128

 CHAPTER

FOUR

NETWORK INFRASTRUCTURE

• NAT-T (page 130)

• Secondary IP (page 155)
• NSG Underlay Border Router (page 165)
• NSG Border Router (page 178)
• Network Uplinks (Wired and LTE) (page 201)

129
 VNS User Guide, Release 5.3.2

4.1 NAT-T

• NAT-T overview (page 131)

– Supported NAT-T features (page 131)
• Supported NAT types (page 131)
– NAT-T compatibility (page 132)
• NAT-T use cases (page 132)
– Dual uplink support (page 132)

* Scenario 1: Single uplink with single NAT (page 132)

* Scenario 2: Dual uplink with single NAT (page 133)
* Scenario 3: Dual uplink with two NATs (page 134)
* Scenario 4: NAT uplink and non-NAT uplink (page 135)
– VSC NAT-T mapping use cases (page 136)

* Scenario 1: Dual uplink, 1:1 and endpoint-independent NAT (page 136)

* Scenario 2: Single uplink, endpoint-independent NAT (page 137)
* Scenario 3: Single uplink, 1:1 NAT (page 138)
• NAT-T NPM probes (page 139)
• UBR-only NAT probes (page 140)
• NAT-T visualization (page 141)
• NAT-T configuration specifics (page 142)
• NAT-T workflow (page 142)
• Procedures for NAT-T (page 143)
– Enabling or disabling NAT-T NPM probes in the domain (page 143)
– Configuring NAT-T settings at the NSG Port level (page 143)
– Configuring the NAT probe interval (page 143)
• NAT-T troubleshooting (page 144)
• NAT-T CLI samples (page 144)
– nat-traversal-mapping (page 145)
– bgp-NAT (page 147)
– natt/show (page 148)
– dump Nuage_Aar_Npmg (page 148)
– analyzer/dump (page 149)
– generator/dump (page 149)
– neighbor/show (page 151)
– dump Nuage_Aar_Xpmg_Dpid (page 151)

4.1. NAT-T 130

 VNS User Guide, Release 5.3.2

– get Static_Config_Neighbor (page 151)

– get Local (page 151)
– dpif/show (page 151)
– ipsec/list-policies (page 153)

4.1.1 NAT-T overview

Nuage VNS Network Address Translation Traversal (NAT-T) allows an NSG deployed behind a NAT device (such as a
firewall, address-sharing Internet gateway, or carrier-grade NAT) to traverse the NAT device and correctly program the
WAN datapath. NAT-T is limited to encapsulated traffic on connections that egress the NSG over an uplink interface.
Key functionality of NAT-T includes:
• Dynamic hole punching between all types of NAT devices.
• Support for multiple NAT mappings in all dual uplink scenarios.
• Continuous probing between endpoints to ensure connectivity.
• UBR-only NAT probes to allow support for all symmetric NAT scenarios.
• Visualization framework to graphically display connection status and performance.

Supported NAT-T features

Nuage VNS currently provides support for the following NAT-T features:
• NAT-T NPM probes (page 139)
• UBR-only NAT probes (page 140)
• NAT-T visualization (page 141)

4.1.2 Supported NAT types

The following NAT-T types are supported:

• None: No NAT-T is required. In this case, no DTLS connections are set up to the VSC.
• 1:1 NAT: In this case, the NSG uplink IP is mapped to a unique public IP.
• Endpoint-independent mapping (full cone NAT): NAT-T is required. Any external host (AnyAddr:AnyPort)
can send to an internal address (iAddr:iPort) by sending traffic to an external address (eAddr:ePort). In this case,
DTLS connections are set up to the VSC so that the public IP and public port mappings for VxLAN and IPsec
can be determined by the VSCs and advertised to all the remote NSGs. There is a DTLS session for VxLAN
and another for IPsec (if enabled).
• Address-restricted NAT: NAT-T with at least one NPM probe for either VxLAN or IPsec is required. An
external host (hAddr:AnyPort) can send to an internal address (iAddr:iPort) by sending packets to a mapped
external address (eAddr:ePort) as long as the internal address has sent packets to the external host before.
• Port-restricted NAT: NAT-T with two NPM probes for both VxLAN and IPsec is required. An external host
(hAddr:hPort) can send to an internal address (iAddr:iPort) by sending packets to a mapped external address
(eAddr:ePort) as long as the internal address has sent packets to the external host before. Additionally, the
external host must send traffic to the specific external port (source port) that the internal host had previously

4.1. NAT-T 131

 VNS User Guide, Release 5.3.2

used to contact the external host. In this context, it is the NPM probes for VxLAN and IPsec that ensure the
internal host has opened up the specific external ports.
• Symmetric NAT: NAT-T with two NPM probes for both VxLAN and IPsec is required. An external host can
send to an internal address only once it has received a packet from that address. Each request from the internal
address is mapped to a unique external address and possibly an external port for each destination.

NAT-T compatibility

All combinations of NAT device pairs are supported by Nuage VNS NAT-T. NAT-T between two symmetric NATs
or between a symmetric NAT and a port-restricted NAT is supported only by using an NSG-UBR as described in
UBR-only NAT probes (page 140). All other NAT-T scenarios are supported using the base NAT-T functionality.
Refer to NAT-T troubleshooting (page 144) for information on troubleshooting if an unsupported NAT-T scenario is
being attempted.

4.1.3 NAT-T use cases

This section describes support for NAT-T deployment scenarios.

Dual uplink support

Nuage VNS supports NAT-T in the following dual uplink scenarios:

Scenario 1: Single uplink with single NAT

In this scenario, a single uplink traverses a single NAT device to two VSCs.

4.1. NAT-T 132

 VNS User Guide, Release 5.3.2

Fig. 4.1: Single uplink with single NAT

Scenario 2: Dual uplink with single NAT

In this scenario, dual uplinks traverse a single NAT device to two VSCs.

4.1. NAT-T 133

 VNS User Guide, Release 5.3.2

Fig. 4.2: Dual uplink with single NAT

Scenario 3: Dual uplink with two NATs

In this scenario, dual uplinks traverse two NAT devices to two VSCs.

4.1. NAT-T 134

 VNS User Guide, Release 5.3.2

Fig. 4.3: Dual uplink with two NATs

Scenario 4: NAT uplink and non-NAT uplink

In this scenario, a dual uplink connects to two VSCs, but only one uplink traverses a NAT device.

Fig. 4.4: NAT uplink and non-NAT uplink

4.1. NAT-T 135

 VNS User Guide, Release 5.3.2

Note: Traffic between NSGs that share a common NAT device is hairpinned through the NAT. This capability must
be supported by the NAT device.

VSC NAT-T mapping use cases

The NSG learns the configured NAT mode for each uplink during the bootstrapping process, and in turn notifies the
VSC. Based on the NAT mode, the VSC determines when and how to generate the NAT-T mappings (also known as
VIF routes) for the NSG.
The NAT mapping information in the VSC is relayed to all other NSGs that the primary VSC is connected to in the
same domain and to all other VSCs and their connected NSGs via BGP.
The EVPN BGP route information includes a NAT indicator, NAT mapping (internal IP:port to translated IP:port), a
virtual next hop (NH) which is the NSGs DatapathID. The NSGs DatapathID is advertised as the subnet route next
hop along with an indicator that it is a virtual NH. Receiving (remote) NSGs will have the subnet information with
virtual NH and the NAT mapping routes. For routing, a lookup for the destination subnet will result in a virtual next
hop route. The NSG will then perform a second lookup of the NAT mapping table for determining the next hop and
route accordingly.
In a 1:1 NAT scenario, the VSC returns the public IP visible from its corresponding connection to the NSG. This
allows the NSG to learn its public IP on that link.

Note: In a 1:1 NAT or endpoint-independent NAT with dual uplink scenario, if all VSCs on an NSG uplink that
requires NAT-T become unavailable in addition to the corresponding NSG uplink port going down, all traffic is black-
holed until all NAT-T routes are withdrawn.

The examples in this section describe how the VSC generates NAT-T mappings in various scenarios.

Scenario 1: Dual uplink, 1:1 and endpoint-independent NAT

On the uplink with the endpoint-independent NAT device, assume that the port hole-punched at the NAT device for
VxLAN traffic is 200 and for IPsec traffic is 2000. VSC1 and VSC2 generate the mappings as listed in the tables
below.

4.1. NAT-T 136

 VNS User Guide, Release 5.3.2

Fig. 4.5: Dual uplink with 1:1 NAT and endpoint-independent or symmetric NAT

The following table lists the mappings for VSC1:

VxLAN IPsec
DPID 50.50.50.50 50.50.50.50
Public IP 203.0.113.2 203.0.113.2
Public port VxLAN port IPsec port
Private IP 10.100.100.100 10.100.100.100
Private port VxLAN port IPsec port
The following table lists the mappings for VSC2:
VxLAN IPsec
DPID 50.50.50.50 50.50.50.50
Public IP 203.0.113.1 203.0.113.1
Public port 200 2000
Private IP 10.200.200.200 10.200.200.200
Private port VxLAN port IPsec port

Scenario 2: Single uplink, endpoint-independent NAT

In this scenario, assume that the port hole-punched at the NAT device for VxLAN traffic is 200 and for IPsec is 2000.

4.1. NAT-T 137

 VNS User Guide, Release 5.3.2

Fig. 4.6: Single uplink NSG with endpoint-independent NAT

The following table lists the mappings:

VxLAN IPsec
DPID 50.50.50.50 50.50.50.50
Public IP 203.0.113.1 203.0.113.1
Public port 200 2000
Private IP 10.200.200.200 10.200.200.200
Private port VxLAN port IPsec port

Scenario 3: Single uplink, 1:1 NAT

If the NAT device is a 1:1 NAT, no DTLS session is initiated.

4.1. NAT-T 138

 VNS User Guide, Release 5.3.2

Fig. 4.7: Single uplink with 1:1 NAT

The following table lists the mappings:

VxLAN IPsec
DPID 50.50.50.50 50.50.50.50
Public IP 203.0.113.2 203.0.113.2
Public port VxLAN port IPsec port
Private IP 10.200.200.200 10.200.200.200
Private port VxLAN port IPsec port

4.1.4 NAT-T NPM probes

When NAT-T is required, the VSC uses two Datagram Transport Layer Security (DTLS) sessions with the NSG. The
local NSG uses information from the VSC to populate a table of address:port NAT-T mappings for both IPsec and
VxLAN. The local NSG returns mapping information to the VSC. The VSC sends this mapping information to all
remote NSGs, so that each remote NSG has full knowledge of the NAT devices and mappings in the domain.
Continuous probing of the DTLS sessions ensures that the NAT mappings do not time out in the NAT device. Traffic
received by the NAT is always forwarded to the private interface on the NSG.
After the local and remote NSGs learn each others’ external address:port mappings from the VSC, they create two
NPM probes with each other — one for IPsec and one for VxLAN. This has the effect of UDP hole punching, and
allows NSGs behind address-restricted, port-restricted, and some instances of symmetric NAT types to communicate.

4.1. NAT-T 139

 VNS User Guide, Release 5.3.2

Fig. 4.8: NPM probe process

The process for establishing DTLS sessions and NPM probes between NSGs behind NAT devices and the VSC is as
follows:
Step 1 NSG-A and NSG-B establish DTLS sessions with the VSC for both IPsec and VxLAN.
Step 2 The NSGs learn the external address:port mappings (IPa’:Pa’ and IPb’:Pb’) for IPsec and VxLAN.
The NSGs send the information to the VSC via the control plane. The VSC forwards mapping
information to all other NSGs behind NAT devices in the domain.
Step 3 NSG-A and NSG-B establish NPM probes for both IPsec and VxLAN.

4.1.5 UBR-only NAT probes

Port-restricted/symmetric and symmetric/symmetric NAT compatibility scenarios do not function with only the NPM
probes configured. Connection between NSGs in these scenarios cannot be established because each NSG may be
blocked by lack of a probe-provided mapping to reach the other NSG. As a result, hole punching fails.
An NSG-UBR can be provisioned to establish connectivity between these two NSGS if hole punching fails for the
direct paths between them. You can provision an NSG-UBR between the NSGs behind NAT devices and enable the
Traffic Through UBR-Only parameter on the NSG Uplink Ports. When the parameter is enabled, the NSG uplink
port is removed from the full mesh of NPM probes and all traffic that would be transported through the local NSG
network uplink directly to the remote NSG is sent via the NSG-UBR instead.

Note: In this scenario, the NSG-UBRs must be part of an NSG-UBR Group.

4.1. NAT-T 140

 VNS User Guide, Release 5.3.2

4.1.6 NAT-T visualization

The visualization feature set can be used to monitor and historically analyze NAT-T reachability between two NSGs.
See the Application-Aware Routing Visualization (page 505) chapter for more information about accessing the visual-
ization feature set.
NAT-T visualization can be viewed from the NAT-T Events dashboard for the selected NSG. The NAT-T Reachability
Status visualization displays a heat map of control and data plane NAT-T reachability. The heat map displays a
chronological progression of NAT-T reachability toward all NSGs with NAT-T enabled. Three different colors are
displayed in the heat map:
• Green - Indicates full bidirectional NAT-T reachability with the destination NSG.
• Yellow - Indicates a change reason for either control or data plane reachability, resulting in missing reachability
information or degraded traffic.
• Red - Indicates no reachability for either the control or data plane.
Hover over a square in the heat map to view the timestamp, destination, and reachability status for the probe interval.
As required, configure the Time Interval and Refresh Interval parameters to specify the recency of the displayed
data and the rate at which it is refreshed.

Fig. 4.9: NAT-T reachability status

Select a square in the heat map to drill down to the Latest x NAT Probe Results and Last x NAT Probe Results at <time>
visualizations. The Latest x NAT Probe Results visualization displays the results of the last NAT probes received from
the selected NSG. The Last x NAT Probe Results at <time> visualization displays the results of the last NAT probes
received from the selected NSG up until the selected time. Both visualizations display the last NAT probes regards
of the path on which they were received. Select a value from the Size dropdown menu to specify how many results
are listed in the chart visualizations. The Change Reason column indicates the reason for the latest change in NAT-T
reachability.

Fig. 4.10: NAT-T probe results

4.1. NAT-T 141

 VNS User Guide, Release 5.3.2

4.1.7 NAT-T configuration specifics

NAT-T is enabled by default. There are two automatically provisioned probe sessions for each pair of NAT neighbors
in the domain. When the probes are enabled and associated with the NAT NPM, the two NSGs can dynamically update
their mapping information. These probes can be disassociated from the NAT NPM at the NSG port level. If the probes
are disassociated, no probes are sent on the uplink and no DTLS sessions are established with the VSC. If the probes
are disassociated, no other probe can be associated with the NAT NPM.

As CSProot, navigate to Platform Configuration -> Infrastructure -> System Configuration -> VNS.
NAT-T NPM probes can be disabled for IPsec and VxLAN by disabling the Add Probe to IPsec NPM and Add Probe
to VxLAN NPM parameters in the system settings. If these parameters are disabled, DTLS sessions are still created,
but probes are not used. Configuring these parameters requires CSPRoot privileges.

Note: Add Probe to VxLAN NPM must be enabled when AAR is enabled for NSGs communicating with another
NSG behind a NAT device.

Navigate to Infrastructure -> Network Services Gateways -> NSG -> Network Port -> Advanced.
The probes can be disassociated from the NAT NPM by disabling the NAT Probes Enabled parameter at the NSG
port level. In a large network with a high number of NSGs, the full mesh of NPM probes may introduce scaling issues.
If scaling issues are encountered, NSGs that are not behind NAT or that are behind 1:1 NAT should have NAT Probes
Enabled set to false.

Note: The NAT Probes Enabled parameter is not available on NSG ports for NSG-UBRs.

You can provision an NSG-UBR between the NSGs behind NAT devices and enable the UBR-Only NAT Probes
parameter on the NSG Uplink Ports. When the parameter is enabled, the NSG uplink port is removed from the full
mesh of NPM probes and all traffic that would be transported through the local NSG network uplink directly to the
remote NSG is sent via the NSG-UBR instead.

As CSProot, navigate to Platform Configuration -> Infrastructure -> Network Service Gateways -> Perfor-
mance Monitors .
The maximum interval at which probes can sent can be configured. The Interval parameter can be configured by
a CSPRoot user on the NAT Probe configuration form. To reduce the risk of scaling issues, the Interval parameter
should be relaxed from the default of 1 packet every 10 seconds to a less frequent rate.

Note: RFC 4787 states that NAT devices should not expire NAT mappings in less than 2 minutes. Therefore it is
possible to set the Interval value to a longer value than the default, such as 1 packet every 30 seconds. Exercise
caution when setting this timer.

Note: You must ensure encryption is enabled at the enterprise level before enabling NAT-T NPM probes. If encryption
is disabled, no NAT-T IPsec probes are sent. See Infrastructure Provisioning (CSPRoot Workflow) (page 23) for more
information about configuring enterprise parameters.

4.1.8 NAT-T workflow

This section provides a high-level workflow of the tasks required to configure VNS NAT-T.

4.1. NAT-T 142

 VNS User Guide, Release 5.3.2

Step 1 Assess your VNS NAT-T requirements based on the type of NAT devices in the network. Refer to
Supported NAT types (page 131) for more information.
Step 2 Enable NAT-T NPM probes in the domain, as required. Refer to Enabling or disabling NAT-T
NPM probes in the domain (page 143) for more information.
Step 3 Configure NAT probes at the NSG port level, as required. Refer to Configuring NAT-T settings at
the NSG Port level (page 143) for more information.
Step 4 Configure NAT probes to be sent via an UBR only, as required. Refer to Configuring NAT-T
settings at the NSG Port level (page 143) for more information.
Step 5 Configure the NAT probe interval, as required. Refer to Configuring the NAT probe interval
(page 143) for more information.
Step 6 Troubleshoot an unsupported NAT-T scenario, if difficulties are encountered. Refer to NAT-T
troubleshooting (page 144) for more information.
Step 7 Monitor NAT-T connectivity and performance. Refer to NAT-T visualization (page 141) for more
information.

4.1.9 Procedures for NAT-T

This section provides the detailed steps required to configure VSD objects to enable the features described in this
chapter.

Enabling or disabling NAT-T NPM probes in the domain

Step 1 As CSProot, navigate to Platform Configuration -> Infrastructure -> System Configuration
-> VNS.
Step 2 Configure the Add Probe to IPsec NPM and Add Probe to VxLAN NPM parameters. These
parameters are enabled by default.

Configuring NAT-T settings at the NSG Port level

Step 1 Navigate to Infrastructure -> Network Services Gateways -> NSG -> Network Port -> Ad-
vanced.
Step 2 Configure the NAT Probes Enabled and UBR-only NAT Probes parameters, as required.

Configuring the NAT probe interval

Step 1 As CSPRoot, Navigate to Platform Configuration -> Infrastructure -> NSG Profiles ->
Performance Monitor .
Step 2 Right-click a NAT probe and select Edit.
Step 3 Configure the Interval parameter and click Update.

4.1. NAT-T 143

 VNS User Guide, Release 5.3.2

4.1.10 NAT-T troubleshooting

Nuage VNS supports NAT-T between two symmetric NATs or between a symmetric NAT and a port-restricted NAT
only if an NSG-UBR is used as described in UBR-only NAT probes (page 140). The generator-dump CLI show
command can be used to determine whether NAT-T is failing due to an unsupported NAT combination. See genera-
tor/dump (page 149).
Perform the following steps to use the generator-dump command to troubleshoot an unsupported NAT combina-
tion:
Step 1 Execute the CLI show command using the following format: vsc# ovs-appctl -t
nuage_perfd generator/dump [UUID]
The command output lists all the generators known to the NSG. The Generator Subtype indicates whether the probes
are IPsec or VxLAN.
Step 2 Observe the Generator State. If the Generator State is “RUNNING”, then the probes are func-
tional. If the Generator State is “CREATED”, it means the NSG has no information for the neigh-
bor flag, or that there are no valid paths available.
Step 3 Observe the path information. If the Path State is “UP”, then the path is valid. The Last Sequence
number parameter increments each time a probe packet is received. If the value is nonzero, then
probe packets are being received.
Step 4 Observe the local and neighbor NSG. If the Enable NAT probes parameters are “True” and the
Traffic Through UBR Only parameters are “False”, then NAT-T probes should be received on the
valid paths.
Step 5 Observe the Start Retry for the paths. If the value is nonzero and incrementing, the generator
probes are continuously sending start control messages without receiving a reply. The Generator
State may be “INITIALIZING”.
If NAT-T probes are expected after verifying the information in these steps, then the NSGs may be attempting to
communicate across an unsupported NAT combination.

Note: It is possible that probes are running on some paths, but not others. This can happen if one of the uplinks is
behind an unsupported NAT combination and the other is not.

4.1.11 NAT-T CLI samples

This section provides sample CLI command outputs relevant to NAT-T.

The following CLI commands are relevant to NAT-T:
• vsc# show vswitch-controller nsg nat-traversal-mapping (page 145)
• vsc# show router bgp routes evpn bgp-NAT (page 147)
• vsc# ovs-appctl natt/show (page 148)
• vsc# ovsdb-client dump Nuage_Aar_Npmg (page 148)
• vsc# ovs-appctl -t nuage_perfd analyzer/dump (page 149)
• vsc# ovs-appctl -t nuage_perfd generator/dump (page 149)
• vsc# ovs-appctl neighbor/show (page 151)
• vsc# ovsdb-client dump Nuage_Aar_Xpmg_Dpid (page 151)

4.1. NAT-T 144

 VNS User Guide, Release 5.3.2

• vsc# redis-cli get Static_Config_Neighbor (page 151)

• vsc# redis-cli get Local (page 151)
• vsc# ovs-appctl dpif/show (page 151)
• vsc# ovs-appctl ipsec/list-policies (page 153)

nat-traversal-mapping

The sample show commands in this section are based on a setup of two NSGs with dual uplinks.
The following table lists information for NSG 1:
Uplink 1 Uplink 2
NAT type No NAT 1:1 NAT
Controller Controller G Controller I
Private IP 10.15.1.254 10.18.1.254
Translated IP n/a 130.104.1.1
The following table lists information for NSG 2:
Uplink 1 Uplink 2
NAT type Address-independent 1:1 NAT
Controller Controller G Controller I
Private IP 10.15.3.254 10.18.3.254
Translated IP 130.104.0.2 130.104.1.2
NAT-Mapping output on controller G:

*vsc# show vswitch-controller nsg nat-traversal-mapping

===============================================================================
NSG NAT Traversal Mappings
===============================================================================
Datapath Id Public Public Private Uplink DTLS Conn
Ip Address Port Ip Address Id Status/Type/
Uptime
-------------------------------------------------------------------------------
57.249.36.216 130.104.0.2 1039 10.15.3.254 0 up/VxLAN/
0d 00:08:23
57.249.36.216 130.104.0.2 1044 10.15.3.254 0 up/IPsec/
0d 00:07:20
119.189.52.131 10.15.1.254 0 10.15.1.254 0 down/VxLAN/
0d 00:00:00
119.189.52.131 10.15.1.254 0 10.15.1.254 0 down/IPsec/
0d 00:00:00
-------------------------------------------------------------------------------
No. of NAT Traversal Mappings: 2
===============================================================================

NAT-Mapping output on controller I:

*vsc# show vswitch-controller nsg NAT-traversal-mapping

===============================================================================
NSG NAT Traversal Mappings
===============================================================================
Datapath Id Public Public Private Uplink DTLS Conn

4.1. NAT-T 145

 VNS User Guide, Release 5.3.2

Ip Address Port Ip Address Id

Status/Type/
Uptime
-------------------------------------------------------------------------------
57.249.36.216 130.104.1.2 0 10.18.3.254 0 down/VxLAN/
0d 00:00:00
57.249.36.216 130.104.1.2 0 10.18.3.254 0 down/IPsec/
0d 00:00:00
119.189.52.131 130.104.1.1 0 10.18.1.254 0 down/VxLAN/
0d 00:00:00
119.189.52.131 130.104.1.1 0 10.18.1.254 0 down/IPsec/
0d 00:00:00
-------------------------------------------------------------------------------
No. of NAT Traversal Mappings: 2
===============================================================================

Note: DTLS will show “down” for both no-NAT case and 1:1 NAT case since there is no DTLS established in these
configurations. It will show “up” only for address-independent case.

The following sample shows NAT mappings on controller G in more detail and specific to a svc-id. Note that the
BGP_VPN entries are those that come from different VSCs and learned via BGP. The NVC entries are the ones that
are local to G.

*vsc# show vswitch-controller nsg NAT-traversal-mapping svc-id 20001

===============================================================================
NSG NAT Traversal Mappings
===============================================================================

------------------------------------------------------------------------------
Legend:
Flag : P -> Primary, S -> Secondary, V -> VxLAN, I -> IPsec
------------------------------------------------------------------------------
Flags Service Datapath Id Public Private Site Owner
Id Ip Address Ip Address Id
Port
------------------------------------------------------------------------------
SV 20001 57.249.36.216 130.104.1.2 10.18.3.254 0 BGP_VPN
4789
PV 57.249.36.216 130.104.0.2 10.15.3.254 0 NVC
1039
PV 119.189.52.131 10.15.1.254 10.15.1.254 0 NVC
4789
SV 119.189.52.131 130.104.1.1 10.18.1.254 0 BGP_VPN
4789
SV 212.196.138.227 10.18.33.254 10.18.33.254 0 BGP_VPN
4789
PV 212.196.138.227 10.15.33.254 10.15.33.254 0 BGP_VPN
4789
SI 57.249.36.216 130.104.1.2 10.18.3.254 0 BGP_VPN
4500
PI 57.249.36.216 130.104.0.2 10.15.3.254 0 NVC
1044
PI 119.189.52.131 10.15.1.254 10.15.1.254 0 NVC
4500
SI 119.189.52.131 130.104.1.1 10.18.1.254 0 BGP_VPN
4500

4.1. NAT-T 146

 VNS User Guide, Release 5.3.2

SI 212.196.138.227
10.18.33.254 10.18.33.254 0 BGP_VPN
4500
PI 212.196.138.227 10.15.33.254 10.15.33.254 0 BGP_VPN
4500
------------------------------------------------------------------------------
No. of NAT Traversal Mappings: 12
------------------------------------------------------------------------------
===============================================================================

bgp-NAT

EVPN BGP-NAT routes with their corresponding next-hops (which are datapath Ids) on controller G.
Tunnel Encap 0 is VxLAN, 1 is IPsec

*vsc# show router bgp routes evpn bgp-NAT

===============================================================================
BGP Router ID:10.20.1.7 AS:1000 Local AS:1000
===============================================================================
Legend -
Status codes : u - used, s - suppressed, h - history, d - decayed, * - valid
Origin codes : i - IGP, e - EGP, ? - incomplete, > - best, b - backup

===============================================================================
BGP EVPN Routes
===============================================================================
Flag RouteType DataPathID Tunnel Public IP:Port
RD NextHop Encap Private IP
As-Path
-------------------------------------------------------------------------------
u*>i NAT 57.249.36.216 0 130.104.1.2:4789
65534:33444 57.249.36.216 10.18.3.254
No As-Path
u*>i NAT 57.249.36.216 1 130.104.1.2:4500
65534:33444 57.249.36.216 10.18.3.254
No As-Path
u*>i NAT 119.189.52.131 0 130.104.1.1:4789
65534:33444 119.189.52.131 10.18.1.254
No As-Path
u*>i NAT 119.189.52.131 1 130.104.1.1:4500
65534:33444 119.189.52.131 10.18.1.254
No As-Path
u*>i NAT 212.196.138.227 0 10.18.33.254:4789
65534:33444 212.196.138.227 10.18.33.254
No As-Path
u*>i NAT 212.196.138.227 1 10.18.33.254:4500
65534:33444 212.196.138.227 10.18.33.254
No As-Path
u*>i NAT 212.196.138.227 0 10.15.33.254:4789
65534:38474 212.196.138.227 10.15.33.254
No As-Path
u*>i NAT 212.196.138.227 1 10.15.33.254:4500
65534:38474 212.196.138.227 10.15.33.254
No As-Path
u*>i NAT 57.249.36.216 0 130.104.1.2:4789
65534:41854 57.249.36.216 10.18.3.254
No As-Path

4.1. NAT-T 147

 VNS User Guide, Release 5.3.2

u*>i NAT 57.249.36.216 1 130.104.1.2:4500

65534:41854 57.249.36.216 10.18.3.254
No As-Path
u*>i NAT 119.189.52.131 0 130.104.1.1:4789
65534:41854 119.189.52.131 10.18.1.254
No As-Path
u*>i NAT 119.189.52.131 1 130.104.1.1:4500
65534:41854 119.189.52.131 10.18.1.254
No As-Path
u*>i NAT 212.196.138.227 0 10.18.33.254:4789
65534:41854 212.196.138.227 10.18.33.254
No As-Path
u*>i NAT 212.196.138.227 1 10.18.33.254:4500
65534:41854 212.196.138.227 10.18.33.254
No As-Path
u*>i NAT 212.196.138.227 0 10.15.33.254:4789
65534:54704 212.196.138.227 10.15.33.254
No As-Path
u*>i NAT 212.196.138.227 1 10.15.33.254:4500
65534:54704 212.196.138.227 10.15.33.254
No As-Path
-------------------------------------------------------------------------------
Routes : 16
===============================================================================

The following show command provides a compact view of the VxLAN and IPsec NAT mappings for each path to a
neighbor. It also indicates whether the VSC-learned or probe-learned mapping is currently active.

natt/show

# ovs-appctl natt/show alubr0 212.196.138.227

Local Remote Tunnel Mapping Used VSC Learnt Mapping Probe
˓→Learnt Mapping

Primary1 Primary1 VxLAN Probe Learnt 130.104.0.1:1043 130.104.

˓→0.5:1753

Primary1 Primary1 IPSec Probe Learnt 130.104.0.1:1040 130.104.

˓→0.5:1730

Secondary2 Primary1 VxLAN Probe Learnt 130.104.0.1:1043 130.104.

˓→0.6:1717

Secondary2 Primary1 IPSec Probe Learnt 130.104.0.1:1040 130.104.

˓→0.6:1702

dump Nuage_Aar_Npmg

vsc# ovsdb-client dump Nuage_Aar_Npmg

Nuage_Aar_Npmg table
_uuid dscp flags grp_id mtu npmg_uuid
˓→ priority rate svc_id
------------------------------------ ---- ----- ------ --- ---------------------------
˓→----------- -------- ------- ------

8da3d72e-d9b6-46d3-9ffa-7ce4bd7ec48f 46 36 0 140 "37d34d4d-b950-49d9-8509-

˓→51d937e23ef1" 0 "0.100" 0
e461f524-7d64-462a-9c7c-e77462f664a3 46 68 0 140 "4eaa6c3c-7de5-42fa-9ba9-
˓→7943f4e95313" 0 "0.100" 0

4.1. NAT-T 148

 VNS User Guide, Release 5.3.2

analyzer/dump

vsc# ovs-appctl -t nuage_perfd analyzer/dump

UUID DPID PATH_
˓→RUNNING PATH_STOPPED

37d34d4d-b950-49d9-8509-51d937e23ef1 200.189.172.56
˓→ 2 0

4eaa6c3c-7de5-42fa-9ba9-7943f4e95313 200.189.172.56
˓→ 2 0

4eaa6c3c-7de5-42fa-9ba9-7943f4e95313 48.121.197.184
˓→ 4 0

37d34d4d-b950-49d9-8509-51d937e23ef1 48.121.197.184
˓→ 4 0

vsc# ovs-appctl -t nuage_perfd analyzer/dump 37d34d4d-b950-49d9-8509-51d937e23ef1

˓→ 200.189.172.56 details
Subtype: VxLAN NAT-T
Path: 0, Current State: Running, Session ID: 4. Latency: 7.953910, Jitter 3.029239,
˓→Pkt Loss: 0.000000, Jitter Miss: 1.000000, Last Sequence Number:718

NAT-T: public IP 130.104.0.5, public port 1131, publish timestamp

˓→Sun, 10 Sep 2017 18:11:25

Path: 1, Current State: Running, Session ID: 4. Latency: 8.124963, Jitter 2.870259,
˓→Pkt Loss: 0.000000, Jitter Miss: 16.000000, Last Sequence Number:718

NAT-T: public IP 130.104.0.6, public port 1179, publish timestamp

˓→Sun, 10 Sep 2017 18:11:25

Path: 2, Current State: Init, Session ID: 0. Latency: 0.000000, Jitter 0.000000, Pkt
˓→Loss: 0.000000, Jitter Miss: 0.000000, Last Sequence Number:0

Path: 3, Current State: Init, Session ID: 0. Latency: 0.000000, Jitter 0.000000, Pkt
˓→Loss: 0.000000, Jitter Miss: 0.000000, Last Sequence Number:0

Jitter Appid List:

Latency Appid List:
PacketLoss Appid List:

generator/dump

vsc# ovs-appctl -t nuage_perfd generator/dump

UUID IP
˓→GENERATOR_STATE PROBE_RUNNING PROBE_STOPPED PROBE_STARTING
˓→ PROBE_STOPPING PATH_VALID

37d34d4d-b950-49d9-8509-51d937e23ef1 200.189.172.56
˓→ RUNNING 2 0 0
˓→ 0 2

4eaa6c3c-7de5-42fa-9ba9-7943f4e95313 48.121.197.184
˓→ RUNNING 4 0 0
˓→ 0 4

4eaa6c3c-7de5-42fa-9ba9-7943f4e95313 200.189.172.56
˓→ RUNNING 2 0 0
˓→ 0 2

4.1. NAT-T 149

 VNS User Guide, Release 5.3.2

37d34d4d-b950-49d9-8509-51d937e23ef1 48.121.197.184
˓→ RUNNING 4 0 0
˓→ 0 4

.. code-block:: none

vsc# ovs-appctl -t nuage_perfd generator/dump 37d34d4d-b950-49d9-8509-51d937e23ef1

˓→ 200.189.172.56 details

Generator State: RUNNING

Generator Subtype: VxLAN NAT-T
Path: 0, Path State: UP, Start Retry: 0, Stop Retry: 0, Probe State: Running, Session
˓→ID: 2, Last Sequence number: 708

Path: 1, Path State: INVALID, Start Retry: 0, Stop Retry: 0, Probe State: Init,
˓→Session ID: 1, Last Sequence number: 0

Path: 2, Path State: UP, Start Retry: 0, Stop Retry: 0, Probe State: Running, Session
˓→ID: 2, Last Sequence number: 708

Path: 3, Path State: INVALID, Start Retry: 0, Stop Retry: 0, Probe State: Init,
˓→Session ID: 1, Last Sequence number: 0

Grp Type: 1
Probe Config Data
PROBE_UUID: 37d34d4d-b950-49d9-8509-51d937e23ef1
Dscp : 46, Rate : 0.100000, Mtu : 140
Local NSG data
Dpid: 138.251.137.52
Uplink Ip: 10.15.3.254,
Priority: 1
Underlay Id: 0
Is UBR Uplink: false
Is Functional: true
Enable NAT Probes: true
Traffic Through UBR Only: false

Uplink Ip: 10.18.3.254,

Priority: 4
Underlay Id: 0
Is UBR Uplink: false
Is Functional: true
Enable NAT Probes: true
Traffic Through UBR Only: false

Neighbor NSG Data

Dpid: 200.189.172.56 Domain:
Uplink Ip: 130.104.0.1,
Priority: 1
Underlay Id: 0
Is UBR Uplink: false
Is Functional: true
Enable NAT Probes: true,
Traffic Through UBR Only: false

Uplink Ip: 0.0.0.0,

Priority: 0
Underlay Id: 0
Is UBR Uplink: false
Is Functional: false

4.1. NAT-T 150

 VNS User Guide, Release 5.3.2

Enable NAT Probes: false,

Traffic Through UBR Only: false

neighbor/show

vsc# ovs-appctl neighbor/show alubr0

Neighbor Ref Nhop1 Nhop2 Nhop3
˓→ Nhop4
200.189.172.56 0 130.104.0.1(p1,0,2) ::::::
48.121.197.184 0 10.15.33.254(p1,0,2) :::: 10.18.33.254(s2,0,2)

dump Nuage_Aar_Xpmg_Dpid

vsc# ovsdb-client dump Nuage_Aar_Xpmg_Dpid

Nuage_Aar_Xpmg_Dpid table
_uuid app_uuid dpid
˓→ oneway_latency_sla_for_nsg xpmg_ovsdb_uuid xpmg_uuid
------------------------------------ -------------------------------------- ----------
˓→ -------------------------- -------------------------------------- -----------------

˓→---------------------

1d9b277f-045a-4aaa-8437-5e75b0b00b2c "00000000-0000-0000-0000-000000000000" 950844872

˓→ 0 "8da3d72e-d9b6-46d3-9ffa-7ce4bd7ec48f" "37d34d4d-b950-
˓→49d9-8509-51d937e23ef1"

aa229ac2-8d05-4bbf-a0a3-2a07345367aa "00000000-0000-0000-0000-000000000000" 950844872

˓→ 0 "e461f524-7d64-462a-9c7c-e77462f664a3" "4eaa6c3c-7de5-
˓→42fa-9ba9-7943f4e95313"

e14a3d54-1415-4e47-a49d-be82abe62b22 "00000000-0000-0000-0000-000000000000"
˓→3099949360 0 "8da3d72e-d9b6-46d3-9ffa-7ce4bd7ec48f"
˓→"37d34d4d-b950-49d9-8509-51d937e23ef1"

5736502e-293c-4f94-9a3f-653c0505db74 "00000000-0000-0000-0000-000000000000"
˓→3099949360 0 "e461f524-7d64-462a-9c7c-e77462f664a3"
˓→"4eaa6c3c-7de5-42fa-9ba9-7943f4e95313"

vsc#

get Static_Config_Neighbor

vsc# redis-cli get Static_Config_Neighbor_200.189.172.56

"200.189.172.56,1:0:1,-1:-1:-1,-1:-1:-1,-1:-1:-1"

get Local

vsc# redis-cli get Local_138.251.137.52

"138.251.137.52,10.15.3.254:1:0:0:1,0.0.0.0,0.0.0.0,10.18.3.254:4:0:0:1,"

dpif/show

4.1. NAT-T 151

 VNS User Guide, Release 5.3.2

vsc# ovs-appctl dpif/show

system@ovs-system: hit:14884 missed:28544

alubr0:
alubr0 65534/1: (internal)
ep1ltep-35f281 32/7: (vxlan: df_default=false, in_
˓→key=3535489, local_ip=10.15.3.254)

ep1ltep-99e0eb 45/7: (vxlan: df_default=false, in_

˓→key=10084587, local_ip=10.15.3.254)

ep1ltep-cb9651 25/7: (vxlan: df_default=false, in_

˓→key=13342289, local_ip=10.15.3.254)

es2ltep-35f281 33/7: (vxlan: df_default=false, in_

˓→key=3535489, local_ip=10.18.3.254)

es2ltep-99e0eb 46/7: (vxlan: df_default=false, in_

˓→key=10084587, local_ip=10.18.3.254)

es2ltep-cb9651 26/7: (vxlan: df_default=false, in_

˓→key=13342289, local_ip=10.18.3.254)

iltep-501b3c 42/13: (vxlan: df_default=false, dst_

˓→port=5770, in_key=5249852, local_ip=138.251.137.52)

iltep-b81dc6 20/13: (vxlan: df_default=false, dst_

˓→port=5770, in_key=12066246, local_ip=138.251.137.52)

iltep-fffffd 18/13: (vxlan: df_default=false, dst_

˓→port=5770, in_key=16777213, local_ip=138.251.137.52)

nuage-bgp 7/11: (internal)

port3.16 8/12: (system)
port4.1 27/15: (system)
port4.10 36/22: (system)
port4.2 28/16: (system)
port4.3 29/17: (system)
port4.4 30/18: (system)
port4.5 31/19: (system)
port4.6 34/20: (system)
port4.7 35/21: (system)
svc-app-tap 5/9: (internal)
svc-dpi-tap 6/10: (internal)
svc-pat-tap 3/4: (internal)
svc-rl-tap1 1/2: (system)
svc-rl-tap2 2/3: (system)
svc-spat-tap 4/5: (internal)
ti38acbdc81f28a368 43/13: (vxlan: df_default=false,
˓→dst_port=5770, key=5249852, local_ip=138.251.137.52, remote_ip=200.189.172.56)

ti38acbdc84e1f 23/13: (vxlan: df_default=false, dst_

˓→port=5770, key=16777213, local_ip=138.251.137.52, remote_ip=200.189.172.56)

ti38acbdc8b6271b9 24/13: (vxlan: df_default=false,

˓→dst_port=5770, key=12066246, local_ip=138.251.137.52, remote_ip=200.189.172.56)

tib8c579301f28a368 39/13: (vxlan: df_default=false,

˓→dst_port=5770, key=5249852, local_ip=138.251.137.52, remote_ip=48.121.197.184)

tib8c579304e1f 17/13: (vxlan: df_default=false, dst_

˓→port=5770, key=16777213, local_ip=138.251.137.52, remote_ip=48.121.197.184)

tib8c57930b6271b9 19/13: (vxlan: df_default=false,

˓→dst_port=5770, key=12066246, local_ip=138.251.137.52, remote_ip=48.121.197.184)

tp38acbdc810068821f28a368409 44/14: (vxlan: df_

˓→default=false, dst_port=1033, key=5249852, remote_ip=130.104.0.1)

tp38acbdc810068824e1f409 21/14: (vxlan: df_

˓→default=false, dst_port=1033, key=16777213, remote_ip=130.104.0.1)

tp38acbdc81006882b6271b9409 22/14: (vxlan: df_

˓→default=false, dst_port=1033, key=12066246, remote_ip=130.104.0.1)

tp38acbdc850068821f28a36846b 48/23: (vxlan: df_

˓→default=false, dst_port=1131, key=5249852, remote_ip=130.104.0.5)

4.1. NAT-T 152

 VNS User Guide, Release 5.3.2

tp38acbdc850068824e1f46b 47/23: (vxlan: df_

˓→default=false, dst_port=1131, key=16777213, remote_ip=130.104.0.5)
tp38acbdc85006882b6271b946b 49/23: (vxlan: df_
˓→default=false, dst_port=1131, key=12066246, remote_ip=130.104.0.5)

tp38acbdc860068821f28a36849b 51/24: (vxlan: df_

˓→default=false, dst_port=1179, key=5249852, remote_ip=130.104.0.6)

tp38acbdc860068824e1f49b 50/24: (vxlan: df_

˓→default=false, dst_port=1179, key=16777213, remote_ip=130.104.0.6)

tp38acbdc86006882b6271b949b 52/24: (vxlan: df_

˓→default=false, dst_port=1179, key=12066246, remote_ip=130.104.0.6)

tpb8c57930fe210f0a1f28a36812b5 41/7: (vxlan: df_

˓→default=false, key=5249852, remote_ip=10.15.33.254)

tpb8c57930fe210f0a4e1f12b5 13/7: (vxlan: df_

˓→default=false, key=16777213, remote_ip=10.15.33.254)

tpb8c57930fe210f0ab6271b912b5 15/7: (vxlan: df_

˓→default=false, key=12066246, remote_ip=10.15.33.254)

tsb8c57930fe21120a1f28a36812b5 40/7: (vxlan: df_

˓→default=false, key=5249852, remote_ip=10.18.33.254)

tsb8c57930fe21120a4e1f12b5 14/7: (vxlan: df_

˓→default=false, key=16777213, remote_ip=10.18.33.254)

tsb8c57930fe21120ab6271b912b5 16/7: (vxlan: df_

˓→default=false, key=12066246, remote_ip=10.18.33.254)

vp1ltep-501b3c 37/7: (vxlan: df_default=false, in_

˓→key=5249852, local_ip=10.15.3.254)

vp1ltep-b81dc6 9/7: (vxlan: df_default=false, in_

˓→key=12066246, local_ip=10.15.3.254)

vp1ltep-fffffd 11/7: (vxlan: df_default=false, in_

˓→key=16777213, local_ip=10.15.3.254)

vs2ltep-501b3c 38/7: (vxlan: df_default=false, in_

˓→key=5249852, local_ip=10.18.3.254)

vs2ltep-b81dc6 10/7: (vxlan: df_default=false, in_

˓→key=12066246, local_ip=10.18.3.254)

vs2ltep-fffffd 12/7: (vxlan: df_default=false, in_

˓→key=16777213, local_ip=10.18.3.254)

dtls-br:
dtls-br 65534/8: (internal)
dtls-tap 65279/6: (internal)
dtls-tap-l 65278/7: (vxlan: in_key=16777214)
t10.10.13.7 65277/7: (vxlan: key=16777214, remote_
˓→ip=10.10.13.7, tos=inherit)

t10.10.14.8 65275/7: (vxlan: key=16777214, remote_

˓→ip=10.10.14.8, tos=inherit)

ipsec/list-policies

vsc# ovs-appctl ipsec/list-policies

IPSec policies:
Local DR Mode: No, Last DR mode entry: - , Last DR mode exit: -
VxLAN src VxLAN dst ESP local ESP remote UDP dport
˓→Remote-DR DR-entry DR-exit Customer-ID VRFs
--------------------------------------------------------------------------------------
˓→---------------------------------------------------------------------

138.251.137.52 200.189.172.56 10.15.3.254 130.104.0.5 1140

˓→No - - 10005 (b6271b9,
˓→1f28a368)

4.1. NAT-T 153

 VNS User Guide, Release 5.3.2

138.251.137.52 200.189.172.56 10.18.3.254 130.104.0.6 1176

˓→No - - 10005 (b6271b9,
˓→1f28a368)

138.251.137.52 48.121.197.184 10.15.3.254 10.15.33.254 4500

˓→No - - 10005 (b6271b9,
˓→1f28a368)

138.251.137.52 48.121.197.184 10.18.3.254 10.15.33.254 4500

˓→No - - 10005 (b6271b9,
˓→1f28a368)

138.251.137.52 48.121.197.184 10.15.3.254 10.18.33.254 4500

˓→No - - 10005 (b6271b9,
˓→1f28a368)

138.251.137.52 48.121.197.184 10.18.3.254 10.18.33.254 4500

˓→No - - 10005 (b6271b9,
˓→1f28a368)

4.1. NAT-T 154

 VNS User Guide, Release 5.3.2

4.2 Secondary IP

• Secondary IP overview (page 155)

• Secondary IP use case (page 156)
• Secondary IP configuration specifics (page 156)
– General considerations (page 156)
– BGP (page 157)

* Routing policy (page 157)

– Uplink connection (page 157)
– Bootstrapping considerations (page 157)
• Secondary IP workflow (page 158)
• Procedures for secondary IP (page 158)
– Configuring a BGP neighbor (page 158)
– Configuring an uplink connection for secondary IP (page 158)
• BGP routing policies (page 158)
– Default BGP export routing policy (page 159)
– Sample BGP export routing policy (page 159)
• Secondary IP CLI samples (page 160)
– ip address show port (page 160)
– nuage-bgp-show summary-all (page 161)
– nuage-bgp-show neighbor (page 161)
– nuage-bgp-show routes (page 162)
– nuage-bgp-show bgp-config (page 163)
– nuage-bgp-show rtm-routes (page 164)

4.2.1 Secondary IP overview

The Nuage VNS secondary IP functionality allows all PE-facing NSGs in a domain to be provisioned with a duplicated
/31 private IP address as well as a unique /32 secondary IP address. Each PE interface is configured with the same
BGP, VLAN, and IP addressing settings. This eliminates the need for a unique IP to reach the NSG, which is useful
in domains where IP addressing space is limited. It also allows for a simplified provisioning model for both the NSGs
and the PE devices.
The unique secondary IP is used for traffic such as OpenFlow and VxLAN. An eBGP or iBGP session is established
on each CE-PE connection and used to advertise the secondary IP to the PE. With eBGP, the PE configures the BGP
next-hop as itself for the /32 interface, allowing the rest of the network to route traffic via the PE. The secondary IP
needs to be assigned only to PE-facing NSGs.

4.2. Secondary IP 155

 VNS User Guide, Release 5.3.2

4.2.2 Secondary IP use case

This section describes the use case for secondary IP deployment in a VNS network.

Fig. 4.11: Secondary IP use case

In the above topology, a branch location uses a single MPLS link and an optional high-speed Internet secondary
link. The same /31 private IP address is used on every CE-PE connection. An eBGP session is established on each
connection between the NSG and PE. A unique /32 secondary IP address is assigned to each NSG that connects to
the PE. The eBGP session is sourced with the /31 private IP address and all other traffic uses the secondary IP as the
source.
With the above configuration, the same /31 private IP address can be used on multiple NSGs in the domain. This
reduces IP addressing space, and also allows for a simplified provisioning model for both the NSGs and the PE
devices.
In this scenario, the NSG can bootstrap via the high-speed Internet link. Alternatively, in an MPLS-only scenario, the
NSG can bootstrap via the activation email with the required IP and BGP information.

4.2.3 Secondary IP configuration specifics

This section describes the specific information required to configure secondary IP on an NSG.

General considerations

Consider the following when configuring secondary IP addresses for NSGs:

• Modifying the secondary IP address post-bootstrapping is service-impacting.
• If a secondary IP address is configured post-bootstrapping, the BGP configuration is sent to the NSG via XMPP.
A reload configuration is required for the secondary IP address to be applied and advertised via the BGP session.
• Only one secondary IP address can be configured on an NSG, regardless of the number of uplinks or BGP
neighbors.

4.2. Secondary IP 156

 VNS User Guide, Release 5.3.2

• Secondary IP addresses must not overlap with other interface IP addresses on the same NSG.
• Secondary IP addresses must not be in the same subnet as the /31 private IP address.
• Deleting the secondary IP address and its BGP neighbor results in an OpenFlow failure.
• Secondary IP addresses are not supported on the NSG-BR or NSG-UBR.
• PAT-to-underlay and NAT-T are not supported for secondary IP addresses.
• Secondary IP addresses cannot be configured at the NSG template level.

BGP

BGP must be enabled on the domain in order for BGP sessions to be established for secondary IP addresses. If an
NSG is configured with a secondary IP address, it must be configured with a BGP neighbor.

Navigate to Infrastructure -> NSG -> Network Port -> VLAN -> BGP Neighbor .

Note: If iBGP is used, the PE may need to be configured with a route reflector.

Routing policy

A BGP export routing policy does not need to be explicitly configured to advertise a secondary IP to an uplink BGP
peer. A default BGP export routing policy is automatically associated with the uplink BGP peer if no other policy is
configured. Refer to Default BGP export routing policy (page 159) to view the default policy.
If a specific BGP export routing policy needs to be associated with an uplink BGP peer with a secondary IP, the policy
should be configured post-bootstrapping.
Configure the following for the export policy for secondary IP:
• Add an entry to the default route advertisements matching on the STATIC protocol with the action reject-route.
• Add an entry to advertise the secondary IP. Match on the NVC protocol with the action accept-route.
Refer to Sample BGP export routing policy (page 159) to view a sample BGP export routing policy.

Uplink connection

Secondary IP addresses are supported only on static uplink connections. Once the Connection is set to Static, the
Secondary IP Address and Associated BGP Neighbor can be specified.

Navigate to Infrastructure -> NSG -> Network Port -> VLAN -> Uplink Connection .

Bootstrapping considerations

If a secondary IP is configured prior to NSG bootstrapping, the /31 private IP address, /32 secondary IP address, and
BGP neighbor configuration is sent to the NSG via the registration email. During bootstrapping, a BGP session is
established with the PE neighbor using this information. Up to three attempts are made to establish the session, with
10 seconds between each attempt. When the BGP session is established and the secondary IP is advertised, WAN
connectivity checks begin. If the BGP session is not established after three attempts, bootstrapping proceeds using the
second uplink, if available. If a second uplink is not available, the BGP session is not established and a failure message
is displayed.

4.2. Secondary IP 157

 VNS User Guide, Release 5.3.2

Note: If a BGP blob is specified pre-bootstrapping, it is not sent as part of the registration email. It is sent via XMPP
post-bootstrapping.

4.2.4 Secondary IP workflow

This section provides a high-level workflow of the tasks required to configure secondary IP addresses.

Note: Modifying the secondary IP address post-bootstrapping is service-impacting and may require a reload config-
uration.

Step 1 Ensure BGP is enabled in the domain. Refer to BGP enabled (page 330) for more information.
Step 2 Configure a BGP neighbor on the PE-facing NSG. Refer to Configuring a BGP neighbor
(page 158) for more information.
Step 3 Specify the secondary IP address and associate the BGP neighbor on the uplink connection. Refer
to Configuring an uplink connection for secondary IP (page 158) for more information.

4.2.5 Procedures for secondary IP

Configuring a BGP neighbor

Step 1 Navigate to Infrastructure -> NSG -> Network Port -> VLAN -> BGP Neighbor .
Step 2 Enter a Name, Description, Peer IP, and Peer AS.
Step 3 If required, specify an Import or Export Routing Policy.
Step 4 Click Update.

Configuring an uplink connection for secondary IP

Step 1 Navigate to Infrastructure -> NSG -> Network Port -> VLAN -> Uplink Connection .
Step 2 Right-click an uplink connection and select Edit.
Step 3 Select Static for the Connection.
Step 4 Enter a Role, **IP Address, Gateway, DNS, and Secondary IP Address.
Step 5 Select an Associated BGP Neighbor and click Update.

4.2.6 BGP routing policies

This section provides BGP export routing policies for secondary IP addresses.

4.2. Secondary IP 158

 VNS User Guide, Release 5.3.2

Default BGP export routing policy

The following default BGP export routing policy is automatically associated with the uplink BGP peer if no other
policy is configured.
policy-statement "_nuage_int_pol_exp_bgpvpn"
entry 10
from
protocol bgp
exit
action accept
exit
exit
entry 20
from
protocol bgp-vpn
exit
action accept
exit
exit
entry 30
from
protocol nvc-local
exit
action accept
exit
exit
entry 40
from
protocol nvc
exit
action accept
exit
exit
default-action reject
exit

Sample BGP export routing policy

The following is a sample BGP export routing policy configured for secondary IP.
<?xml version="1.0" encoding="UTF-8"?>
<routing-policy xmlns="alu:nuage:bgp:routing:policy">
<policy-definition>
<statements>
<statement>
<name>entry_1</name>
<conditions>
<install-protocol-eq>STATIC</install-protocol-eq>
</conditions>
<actions>
<reject-route />
</actions>
</statement>
<statement>
<name>entry_2</name>
<conditions>

4.2. Secondary IP 159

 VNS User Guide, Release 5.3.2

<install-protocol-eq>NVC</install-protocol-eq>
</conditions>
<actions>
<accept-route-set>
<accept-route />
</accept-route-set>
</actions>
</statement>
</statements>
<default-action>
<accept-route-set>
<accept-route />
</accept-route-set>
</default-action>
</policy-definition>
</routing-policy>

4.2.7 Secondary IP CLI samples

This section provides sample CLI command outputs relevant to secondary IP.
The following CLI commands are relevant to secondary IP:
• ip address show port (page 160)
• nuage-bgp-show summary-all (page 161)
• nuage-bgp-show neighbor (page 161)
• nuage-bgp-show routes (page 162)
• nuage-bgp-show bgp-config (page 163)
• nuage-bgp-show rtm-routes (page 164)

ip address show port

Note: In the following sample, 1.1.1.1/32 is the secondary IP and 193.168.1.1/32 is private IP with a /31 address. In
the output, the private IP may be shown as a /32 address due to Linux limitations.

# tools vswitch 1.1.1.1 command "ip address show port1"

12: port1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc htb state UP qlen 1000
link/ether 00:90:0b:3a:b5:08 brd ff:ff:ff:ff:ff:ff
inet 1.1.1.1/32 brd 1.1.1.1 scope global port1
valid_lft forever preferred_lft forever
inet 193.168.1.1/32 brd 193.168.1.1 scope global port1
valid_lft forever preferred_lft forever

# tools vswitch 1.1.1.1 command "ifconfig port1"

port1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 1.1.1.1 netmask 255.255.255.255 broadcast 1.1.1.1
ether 00:90:0b:3a:b5:08 txqueuelen 1000 (Ethernet)
RX packets 1743 bytes 302347 (295.2 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1739 bytes 267987 (261.7 KiB)

4.2. Secondary IP 160

 VNS User Guide, Release 5.3.2

TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

device memory 0xdfd00000-dfd7ffff

nuage-bgp-show summary-all

In the following sample, the peer IP address for BGP session 193.168.1.0.

*A:vm1# tools vswitch 1.1.1.1 command "nuage-bgp-show summary-all"

===============================================================================
BGP Summary
===============================================================================
Neighbor
ServiceId AS PktRcvd InQ Up/Down State|Rcv/Act/Sent (Addr Family)
PktSent OutQ
-------------------------------------------------------------------------------

193.168.1.0
Svc: 2 200 32 0 00h13m33s 8/8/3 (IPv4)
33 0
-------------------------------------------------------------------------------
value = 0 = 0x0

nuage-bgp-show neighbor

The following sample shows apecific BGP peer information using nuage-bgp-show neighbor <service
id> <peer ip>.
# tools vswitch 1.1.1.1 command "nuage-bgp-show neighbor 2 193.168.1.0"

===============================================================================
BGP Neighbor
===============================================================================
-------------------------------------------------------------------------------
Peer : 193.168.1.0
Group : 193.168.1.0
-------------------------------------------------------------------------------
Peer AS : 200 Peer Port : 52241
Peer Address : 193.168.1.0
Local AS : 200 Local Port : 179
Local Address : 193.168.1.1
Peer Type : Internal
State : Established Last State : Established
Last Event : recvKeepAlive
Last Error : Cease (Connection Collision Resolution)
Local Family : IPv4
Remote Family : IPv4
Hold Time : 90 Keep Alive : 30
Min Hold Time : 0
Active Hold Time : 90 Active Keep Alive : 30
Cluster Id : None
Preference : 170 Num of Update Flaps : 0
Recd. Paths : 2
IPv4 Recd. Prefixes : 8 IPv4 Active Prefixes : 8
IPv4 Suppressed Pfxs : 0 VPN-IPv4 Suppr. Pfxs : 0

4.2. Secondary IP 161

 VNS User Guide, Release 5.3.2

VPN-IPv4 Recd. Pfxs : 0 VPN-IPv4 Active Pfxs : 0

Mc IPv4 Recd. Pfxs. : 0 Mc IPv4 Active Pfxs. : 0
Mc IPv4 Suppr. Pfxs : 0 IPv6 Suppressed Pfxs : 0
IPv6 Recd. Prefixes : 0 IPv6 Active Prefixes : 0
VPN-IPv6 Recd. Pfxs : 0 VPN-IPv6 Active Pfxs : 0
VPN-IPv6 Suppr. Pfxs : 0 L2-VPN Suppr. Pfxs : 0
L2-VPN Recd. Pfxs : 0 L2-VPN Active Pfxs : 0
MVPN-IPv4 Suppr. Pfxs: 0 MVPN-IPv4 Recd. Pfxs : 0
MVPN-IPv4 Active Pfxs: 0 MDT-SAFI Suppr. Pfxs : 0
MDT-SAFI Recd. Pfxs : 0 MDT-SAFI Active Pfxs : 0
FLOW-IPV4-SAFI Suppr*: 0 FLOW-IPV4-SAFI Recd.*: 0
FLOW-IPV4-SAFI Activ*: 0 Rte-Tgt Suppr. Pfxs : 0
Rte-Tgt Recd. Pfxs : 0 Rte-Tgt Active Pfxs : 0
Evpn Suppr. Pfxs : 0 Evpn Recd. Pfxs : 0
Evpn Active Pfxs : 0
Backup IPv4 Pfxs : 0 Backup IPv6 Pfxs : 0
Mc Vpn Ipv4 Recd. Pf*: 0 Mc Vpn Ipv4 Active P*: 0
Backup Vpn IPv4 Pfxs : 0 Backup Vpn IPv6 Pfxs : 0
Input Queue : 0 Output Queue : 0
i/p Messages : 47 o/p Messages : 48
i/p Octets : 1016 o/p Octets : 1031
i/p Updates : 2 o/p Updates : 3
TTL Security : Disabled Min TTL Value : n/a
Graceful Restart : Disabled Stale Routes Time : n/a
Advertise Inactive : Disabled Peer Tracking : Disabled
Advertise Label : None
Auth key chain : n/a
Disable Cap Nego : Disabled Bfd Enabled : Disabled
Flowspec Validate : Disabled Default Route Tgt : Disabled
Local Capability : RtRefresh MPBGP 4byte ASN
Remote Capability : RtRefresh MPBGP 4byte ASN
Local AddPath Capabi*: Disabled
Remote AddPath Capab*: Send - None
: Receive - None
Import Policy : None Specified / Inherited
Export Policy : _nuage_int_pol_exp_bgpvpn

-------------------------------------------------------------------------------
Neighbors : 1
===============================================================================
* indicates that the corresponding row element may have been truncated.
value = 0 = 0x0

nuage-bgp-show routes

# tools vswitch 1.1.1.1 command "nuage-bgp-show routes 2"

===============================================================================
BGP Router ID:57.20.149.193 AS:200 Local AS:200
===============================================================================
Legend -
Status codes : u - used, s - suppressed, h - history, d - decayed, * - valid
Origin codes : i - IGP, e - EGP, ? - incomplete, > - best, b - backup

===============================================================================
BGP IPv4 Routes
===============================================================================

4.2. Secondary IP 162

 VNS User Guide, Release 5.3.2

Flag Network LocalPref MED

Nexthop Path-Id VPNLabel
As-Path
-------------------------------------------------------------------------------
u*>i 10.10.10.0/24 100 125
10.10.201.2 None -
No As-Path
u*>i 10.10.106.0/24 100 125
10.10.201.2 None -
No As-Path
u*>i 10.10.107.0/24 100 125
10.10.201.2 None -
No As-Path
u*>i 10.10.108.0/24 100 125
10.10.201.2 None -
No As-Path
u*>i 10.10.109.0/24 100 125
10.10.201.2 None -
No As-Path
u*>i 10.10.110.0/24 100 35
10.10.201.2 None -
No As-Path
u*>i 10.10.111.0/24 100 35
10.10.201.2 None -
No As-Path
u*>i 10.20.1.6/32 100 125
10.10.201.2 None -
No As-Path
-------------------------------------------------------------------------------
Routes : 8
===============================================================================
value = 0 = 0x0

nuage-bgp-show bgp-config

# tools vswitch 1.1.1.1 command "nuage-bgp-show bgp-config 2"

autonomous-system 200
router-id 57.20.149.193
route-distinguisher 257:257
vrf-target target:16842752:257
vrf-import _nuage_int_pol_uimp
vrf-export _nuage_int_pol_uexp_2

bgp
export "_nuage_int_pol_exp_bgpvpn"
local-as 200
router-id 57.20.149.193
group "193.168.1.0"
local-as 200
neighbor 193.168.1.0
connect-retry 120
keepalive 30
hold-time 90
min-route-advertisement 30
local-as 200
peer-as 200

4.2. Secondary IP 163

 VNS User Guide, Release 5.3.2

split-horizon
exit
exit
no shutdown
exit
value = 0 = 0x0

nuage-bgp-show rtm-routes

# tools vswitch 1.1.1.1 command "nuage-bgp-show rtm-routes 2"

===============================================================================
Route Table (Router: Vrf 2)
===============================================================================
Dest Prefix[Flags] Type Proto Age Pref
Next Hop[Interface Name] Metric
-------------------------------------------------------------------------------
0.0.0.0/0 Remote Static 02h28m22s 255
Ukwn Interface 0
1.1.1.1/32 Local NVC 02h28m22s 0
0
10.10.10.0/24 Remote BGP 02h26m32s 170
0
10.10.106.0/24 Remote BGP 02h26m32s 170
0

4.2. Secondary IP 164

 VNS User Guide, Release 5.3.2

4.3 NSG Underlay Border Router

• Overview (page 165)

• Configuration Objects (page 166)
• Routing via NSG-UBR (page 167)
– Forwarding Behavior Scenarios (page 168)
– Advanced ACL Uplink Preference Scenarios (page 169)
– Underlay Reachability (page 169)
– Service Chaining (page 169)
• NSG-UBR Configuration (page 169)
– Define Underlays (page 170)
– Define Performance Monitor (page 170)
– Instantiate NSG-UBR (page 170)
– Add and/or Extend VLANs (page 171)
– Define Uplink Connections (page 171)
– Configure BFD (page 172)
– Create NSG-UBR Groups (page 172)
– Create NSG Groups (page 173)
– Create NSG Group to UBR Group Bindings (page 173)
– Activate the NSG-UBR (page 174)
• Border Router Support on NSG-UBR (page 174)
• Statistics (page 175)
• Sample BFD Peer Configuration (page 175)
– 7750 SR Sample (page 175)
– Cisco Sample (page 175)
• CLI Show Command Sample (page 176)
– BFD CLI commands (page 176)
– UBR routes CLI command (page 177)

4.3.1 Overview

Underlay Border Router (UBR) feature enables seamless connectivity between NSGs in disjoint underlay networks.
The feature introduces a new NSG personality type, referred to as NSG-UBR, that allows a single VNS domain to
span multiple underlays without having to link the domains. NSG-UBR is supported on NSG-X and NSG-V form
factors. In addition, NSG-UBR supports multi-tenancy.
The UBR feature utilizes the construct of underlay tags, that are assigned to NSG uplinks, to identify the underlays.
Between any source and destination NSGs, a path via direct attached underlays is always preferred. NSG-UBR is used

4.3. NSG Underlay Border Router 165

 VNS User Guide, Release 5.3.2

only as a last resort when the end-points are in disjoint underlays.

As shown in the figure below, NSG-1 will communicate directly with NSG-2 over the common underlay. However,
NSG-1 and NSG-3 will have to use the path via UBR1 as an alternate path.

Fig. 4.12: Underlay Border Router Connectivity

4.3.2 Configuration Objects

The following objects are utilized in the workflows for provisioning an NSG-UBR. The details are described later in
this chapter.
• Uplink Connections include the following contexts:
– Control Uplink is the traditional network uplink used to connect to the controller(s) in the VSC profile,
as well as for bootstrapping.
– Underlay Uplink, is the network uplink that is used only to provide connectivity between NSGs in disjoint
underlays. There is no associated VSC profile. Underlay Uplinks can only be configured on an NSG-UBR.
• Underlay Connection VLAN is a VLAN attribute for configuring an uplink connection on an NSG-UBR as
either a control uplink or underlay uplink. To provision a Control Uplink, the attribute must be disabled (default)
and a VSC profile must be attached. To provision an Underlay Uplink, the attribute must be enabled.
– On an NSG-UBR, one physical port (of type Network) may have a maximum of 1 Control Uplink, but
can have several Underlay Uplinks. This is configured by creating multiple tagged VLANs, of which one
VLAN will be configured as Control Uplink and remaining VLANs as Underlay Uplinks.
– On a NSG-UBR, a maximum of 2 ports may be configured for underlay connections.
– On a regular NSG, on a physical network port, there is only one VLAN (tagged or untagged) per uplink.
• Underlay: Serves as an identification of provider networks which are completely disjointed from other networks
(e.g. VPRNs, MPLS networks, IPv4 networks, IPv6 networks). The underlay object will serve as a way for the
operator to associate specific underlay networks to control uplink and underlay uplinks. For underlay uplinks it
is mandatory to associate an underlay.
• NSG Group: Is a group of NSGs. Created, modified, deleted by CSPRoot and Enterprise Administrator.
• NSG-UBR Group: Is a group of NSG-UBRs with similar attributes, such as geographic proximity. Created by
CSPRoot only but can be used by Enterprise Admininstrator. Referred to as UBR Group for readability.
• UBR Group Binding: Links an NSG Group to one or more NSG-UBR Groups with a preferred priority for
each NSG-UBR group.

Note:
• The UBR Group Bindings must exist for NSGs in disjoint underlays to communicate with each other.

4.3. NSG Underlay Border Router 166

 VNS User Guide, Release 5.3.2

• Traffic between NSGs in disjoint underlays, with UBR bindings, will be over the underlay uplinks of the via
NSG-UBR.
• If regular vPorts are configured on the NSG-UBR, then traffic to other NSGs (that are not in any NSG group)
and are on the same underlay as the NSG-UBR’s control uplink, will be routed via the NSG-UBR’s control
uplink. In other words, the NSG-UBR will operate as a regular NSG in such a scenario.
• An NSG, that is part of an NSG group, cannot have common underlay with the control uplink of an NSG-UBR.

Note: Following are the qualified setup recommendations:

• 6 UBR Group bindings per NSG group
• 8 NSG-UBRs per UBR Group

4.3.3 Routing via NSG-UBR

On an NSG, when a prefix points to a destination NSG on the same underlay, then the next hop is set to the destination
NSG’s DatapathID (or as virtual next hop for NAT-Traversal). When a prefix points to an NSG that is on another
underlay, then the underlay identifier and UBR bindings are used to determine an available NSG-UBR for routing the
traffic. A via NSG UBR has underlay connection VLANs in both underlays of the source and destination NSGs.
If a NSG-UBR is available (defined later in this chapter), then it gets advertised to the NSGs that have bindings to it.
When the source NSG has to send traffic to a destination NSG on a different underlay, then if a via NSG-UBR is found
then:
• The via NSG-UBR’s DatapathID will be used as the next hop and the appropriate underlay (based on the binding)
will be used for forwarding.
• On the receiving NSG-UBR, the next hop will be set to the destination NSG’s DatapathID (or as virtual next
hop for NAT-Traversal). The NSG-UBR will select the underlay on which the destination NSG is reachable.
If there are two or more via NSG-UBRs, then selection is based on one of the following options:
• Priority based UBR Group selection per NSG Group, with ECMP to all the available NSG-UBRs in the highest
priority UBR Group. A UBR Group is not available if every NSG-UBR in the group is not available..
• Performance based UBR Group selection per NSG based on configured priority and underlay path performance
and reachability metrics. Performance monitors are enabled to probe the underlay performance. A UBR Group
is considered to be unavailable if every NSG-UBR in the group is either offline, not reachable (100% packet
loss) or out of SLA. The best NSG-UBR is selection is evaluated every 5 minutes or when there is 100% packet
loss or when there is an SLA violation. Once a path is switched, it is not switched again within 5 seconds of the
switchover.

Note: If an NSG belongs to an NSG group that is associated with two UBR groups, of which only the lower priority
UBR group has an associated performance monitor, AAR path selection is preferred. The NSG routes traffic to an
AAR-selected UBR over a UBR from a higher priority UBR group that does not have an associated performance
monitor.

Since the underlay selection is local, the forward and return path may not necessarily be symmetric.

4.3. NSG Underlay Border Router 167

 VNS User Guide, Release 5.3.2

Forwarding Behavior Scenarios

When source and destination NSGs are in same underlay, then traffic flows directly between NSGs and does not go
via any NSG-UBR. So all the scenarios below assume that source and destination NSGs are in disjoint underlays, and
applies to VxLAN/IPsec traffic with/without NAT-T.
Scenario 1: One UBR Group of all NSG-UBRs, one NSG Group of all NSGs, and no performance monitors.
• If no via NSG-UBR is found, then traffic is dropped at source.
• If only one via NSG-UBR is found, then traffic flows via that NSG-UBR.
• If more than one via NSG-UBR is found, then traffic is routed using per flow ECMP to all via NSG-UBRs.
Scenario 2: All NSG-UBRs have underlay connection VLANs in all underlays, all NSG-UBRs configured in two
UBR Groups with different priorities, one NSG Group of all NSGs, and no performance monitors.
• Traffic is routed using per flow ECMP to all available NSG-UBRs in the UBR group with higher priority.
• If all NSG-UBRs in higher priority group are unavailable, then the group with the lower priority is selected.
Scenario 3: Topology is shown in the Figure below. Performance monitors are associated with UBRGrp1 and UBR-
Grp2.

Fig. 4.13: Example Topology for Performance-Based NSG-UBR Path Selection

The following behavior is observed when NSG-2 initiates traffic to NSG-4 (both NSGs have primary and secondary
uplinks) using performance based path selection:
1. There are one way performance monitor probes from all NSGs to all NSG-UBRs, and from NSG-UBRs to the
NSGs.
2. NSG-2 will select UBRGrp1 based on priority.
3. Within UBRGrp1, NSG-2 will select the path to NSG-UBR1 or NSG-UBR2 via U1 or U2. Assume that delay
NSG-UBR1-U2 < NSG-UBR1-U1 < NSG-UBR2-U1 < NSG-UBR-U2. NSG2 will select NSG-UBR1 via U2.

Note: If a delay attribute is configured then it will be used to flag an SLA violation. Only gateways within SLA will
be eligible for path selection.

4. NSG-UBR1 will select the best path to NSG-4 via U3 or U4. If delay NSG-UBR1-U4 < NSG-UBR1-U3, then
path via U4 will be selected.

4.3. NSG Underlay Border Router 168

 VNS User Guide, Release 5.3.2

Advanced ACL Uplink Preference Scenarios

Assume that the NSG has dual uplinks tagged as primary and secondary. The following uplink preference options are
supported:
• None or Default
• Primary:Secondary
• Primary only
For None, Default, and Primary:Secondary, the common underlay is always preferred between a local and remote
NSG. If traffic needs to go via an NSG-UBR, uplink priority is used. In the example above, NSG-2 forwards via U1
to NSG-UBR1, which forwards via U3 to NSG-4 using the primary uplinks as preferred paths.
For Primary only, uplink priority is preferred over the common underlay.
Refer to the VNS Release Notes for qualification of other uplink preference options.

Underlay Reachability

Bidirectional forwarding detection (BFD) is supported as an option for NSG-UBR underlay reachability verification
(page 60), which is configured as an attribute of the Uplink Connection (page 55) configuration. The BFD configura-
tion workflow (page 171) includes configuration of the the BFD session attributes and the peer in the underlay to which
the session is established. The underlay uplink is considered to be available for routing only if the BFD session is up.
The capability ensures that traffic is not forwarded via an underlay uplink if it is not available. If the BFD session fails,
the routes through the uplink connection are withdrawn and rerouted.

Service Chaining

The following is supported on NSG-UBR:

• Advanced forward to NSG-UBR vPort which enables service chaining from the NSG in the branch site to an
NSG-UBR in a deployment scenario with disjoint underlays.
• Advanced forward to NSG in disjoint underlay via NSG-UBR which enables service chaining from a branch
site NSG to another branch site NSG in a deployment with disjoint underlays, where the datapath between the
sites is enabled via the NSG-UBR.
For workflow details refer to the Service Chaining (page 295) chapter which describes all the service chaining use
cases in VNS.

4.3.4 NSG-UBR Configuration

To deploy an NSG-UBR, CSPRoot needs to:

• Define Underlays (page 170)
• Define Performance Monitors (page 170)
• Instantiate the NSG-UBR (page 170)
• Add and/or Extend VLANs (page 171)
• Define Uplink Connections (Control Uplinks and Underlay Uplink) (page 171)
• Create UBR Groups (page 172)
• Create NSG Groups (page 173)

4.3. NSG Underlay Border Router 169

 VNS User Guide, Release 5.3.2

• Define NSG Group to UBR Group Bindings (page 173)

• Activate the NSG-UBR (page 174)
All the workflows below are in the CSPRoot context.

Define Underlays

Underlay objects are used to identify the different underlay networks in the domain.

1. Navigate to Infrastructure -> Network Service Gateway -> Underlays .

2. Click on towards the bottom of the screen to define a new Underlay.

3. In the New Underlay popup, enter a Name and Description (e.g. “IPv4 underlay” or “MPLS1 underlay”) to
identify the underlay.
CSPRoot may modify and delete underlay objects.
As explained later in this workflow, the underlay must be specified as part of the Uplink Connection configuration of
a network port/VLAN. If not specified, then underlay 0 (i.e. any underlay) is assumed.

Note: For an NSG that is to be added to a NSG Group (for UBR Group binding), all uplink connections must have a
unique (non-zero) underlay assigned.

Define Performance Monitor

Performance Monitor denotes the characteristics of network traffic such as payload size, service class and traffic rate,
and is used to initiate the test traffic with the specified profile for determining underlay path performance.

1. Navigate to Infrastructure -> Network Service Gateway -> Performance Monitor .

2. Click on towards the bottom of the screen to define a new Underlay.

3. Enter the Performance Monitor attributes (page 458) as described in the workflow for application-aware routing.

Note:
• The performance monitor created at the CSPRoot level must be associated to a NSG-UBR group (page 172) by
the CSPRoot.
• A performance monitor that is associated with a NSG-UBR group cannot be deleted.

Instantiate NSG-UBR

This workflow assumes that the CSPRoot has already created a Gateway Template (page 38) with personality NSG-
UBR and required network ports.
1. Select a Gateway Template with personality type NSG-UBR and right-click to select instantiate option. The
CSPRoot can also instantiate using the same workflow (page 51) as described for Enterprise Administrator.
2. The CSPRoot must assign an Installer if automatic email notification is enabled. Otherwise, that association
can be done later. To bootstrap the UBR, an installer must have been selected (required for e-mail registration
and for bootstrapping).

4.3. NSG Underlay Border Router 170

 VNS User Guide, Release 5.3.2

The instantiated NSG-UBRs are listed (in the VSD UI) under a category called Underlay Border Routers, and may
be viewed by navigating to Infrastructure -> Network Service Gateway .

Add and/or Extend VLANs

If port/VLANs have been defined at template level, they will be inherited at the instance level. The NSG-UBR
instance will not auto-create any VLANs if they are not defined at template level.
NSG-UBR must have as many VLANs on the network port(s) as the number of Control and/or Underlay Uplinks
required. For instance, for a single uplink NSG-UBR that is connected to three underlays, four VLANs will be
required; one for control connection and three for underlay connections.

1. Navigate to Infrastructure -> Network Service Gateway -> NSG-UBR instance -> Network Port.
2. Create VLANs on the selected network port. Enable the Underlay Connection VLAN flag if the VLAN will
be used as an underlay uplink. Disable the flag for control uplink.

Note:
• NSG-UBR supports tagged and untagged VLANs. However, a mix of tag and untagged VLANs cannot be on a
single Port
• One of the VLANs must be associated with a VSC profile for the Control Uplink. Otherwise activation of
NSG-UBR will result in an error. Control Uplink must be defined on the same VLAN that is associated with a
VSC profile.
• If the VLAN associated with a VSC profile is untagged, then no Underlay Uplinks are allowed on the same
physical port.
• There is a maximum of one Control Uplink per physical network port. The NSG-UBR can have up to two
Control Uplinks.
• Currently there is no limit on the number of Underlay Uplinks (subject to the VLAN limit on the port). Refer to
the scale section for qualified scenarios.

Define Uplink Connections

Refer to Uplink Connection (page 55) to define the connections for Control Uplink and Underlay Uplink(s). Each
uplink connection will be associated with an underlay.

Attention: For NSG-UBR

• For Underlay Uplinks, connection Mode is Static and Role is None.
• For Control Uplinks, default is DHCP for Mode, and Primary for Role (unless a primary is already defined).

For underlay uplinks, if you select BFD for the Reachability Verification parameter, then you must:
• Configure a BFD session and assign it to the uplink connection as described below.
• Configure the BFD peer as shown in the sample configuration (page 175).

4.3. NSG Underlay Border Router 171

 VNS User Guide, Release 5.3.2

Configure BFD

1. Click on in BFD Session panel.

2. In the popup, configure the following:
• Destination IP Address: Specifies the underlay peer to which the session is established.
• Timer (ms): Specifies the time interval, in milliseconds, between BFD messages.
• Multiplier: Specifies how many consecutive failed BFD messages must occur before the BFD session is de-
clared down.
• Enable Multi-hop: Specifies whether the destination IP is a multi-hop path.
Refer to BFD CLI commands (page 176) to view information on the BFD session. The Local Session State specifies
whether the BFD session is up or down.

Note:
• You can configure only one BFD session per uplink connection.
• BFD configuration only supported at the instance level.
• You must delete the BFD Session object before changing the Reachability Verification parameter to something
other than BFD.

Create NSG-UBR Groups

1. Navigate to Infrastructure -> Network Service Gateway -> UBR Groups .

2. Click on to add a new group.

3. In the popup, enter Name and Description for the NSG-UBR Group.
4. Optionally, associate a Performance Monitor that has been created at the CSPRoot level.
5. Click Create, and the group will be listed in the NSG UBR Groups panel.

4.3. NSG Underlay Border Router 172

 VNS User Guide, Release 5.3.2

6. Select the newly created UBR group, and in the Associated NSG UBRs panel, add NSG-UBR members to the
group using the paperclip icon at the bottom of the panel.
CSPRoot may modify UBR Group by adding/removing NSG-UBRs, or delete UBR Group.

Note:
• Only NSG-UBRs may be in a UBR group.
• Deletion of UBR Group is not allowed if:
– Any NSG-UBR is in the group
– Any NSG Group bindings to the UBR group exist
• Deletion of NSG-UBR from UBR group will display a warning for confirmation if there are bindings to the
UBR Group.
• It is recommended that NSG-UBRs be added to the NSG-UBR group before any bindings to NSG groups are
created.

Create NSG Groups

1. Navigate to Infrastructure -> Network Service Gateway -> NSG Groups .

2. Click on to add a new group.

3. In the popup, enter Name and Description for the NSG Group. Once created, it will be listed in the NSG
Groups panel.
4. Select the newly created NSG group, and select the NSGs tab which will display Associated NSGs panel. Add
NSG members to the group using the paperclip icon at the bottom of the panel.
5. The member NSGs will appear in the Associated NSGs panel.
CSPRoot may modify NSG Group by adding/removing NSGs, or delete NSG Group. Enterprise administrators may
also create NSG groups, and for those that they create, the administrators can add/remote NSGs or delete the group.

Note:
• Only NSG may be included in an NSG group.
• For an NSG-RG, it is recommended that:
– both NSGs (of the RG pair) be part of the NSG group.
– the Uplink Underlays of the RG pair be consistent.
• A new NSG may be added to a group after bindings to UBR groups have been created.
• Deletion of NSG group is not allowed if there are NSGs or NSG BRs in the group.

Create NSG Group to UBR Group Bindings

1. Navigate to Infrastructure -> Network Service Gateway -> NSG Groups . Select an NSG Group.

4.3. NSG Underlay Border Router 173

 VNS User Guide, Release 5.3.2

2. Select the UBR Groups tab (in the panel to the right of the NSG Groups panel) which will display a UBR
Groups panel. Any existing NSG Group to UBR Group bindings will be displayed in this panel.

3. To create new bindings, click on in the panel.

4. In the New UBR Group Binding popup,

• select to link to a UBR group

• optionally, assign a Priority to the binding
• specify a One Way Delay threshold in ms. If the feature is to be disabled, then a value of -1 should be
entered.

Note:
• Maximum of 6 UBR Group bindings on an NSG Group are allowed.
• There can be only one binding between a NSG Group and UBR Group pair.
• Within an NSG Group, UBR Group binding priorities must be unique
• When creating a binding between NSG Group and UBR Group, ensure that all the underlay instances used
by the NSGs in the group are available in the UBR Group (i.e. at least one NSG-UBR in the UBR Group is
associated with the underlays)
• When deleting a binding between NSG Group and UBR Group, which has members present, will generate a
confirmation warning.

Activate the NSG-UBR

Single factor and two factor authentication are supported on the NSG-UBR (Auto-Bootstrapping capability is cur-
rently not supported).
The CSPRoot must activate (page 64) the form factor so that it is bootstrapped and ready for use.

4.3.5 Border Router Support on NSG-UBR

The NSG-UBR also supports regular Border Router (page 178) capabilities for domain linking between WAN and
Data Center.

Fig. 4.14: Border Router Support on NSG-UBR

Any access port VLAN on the NSG-UBR can be cast as a BR-Port by adding BR connections (page 186) and defining
Demarcation service per the workflow for NSG-BR (page 178).

Note:

4.3. NSG Underlay Border Router 174

 VNS User Guide, Release 5.3.2

• The NSG-UBR, on which the BR-Port is created, must be part of a UBR- Group with bindings to an NSG
Group with the linked branch domain.
• If redundant domain links are created from a branch domain to DC domain with each link using a different
BR-Port, then traffic from branch to DC will be ECMP’d. The binding priorities will be ignored.
• NSG-UBR to NSG-UBR traffic, using underlay uplinks, is not supported.

4.3.6 Statistics

Refer to Uplink Statistics (page 218) section for NSG-UBR statistics for control uplinks and underlay uplinks.
In addition, for NSG-UBR, when BR connections are present, statistics are also available at Access port/VLAN level.
The workflow for accessing BR VLAN statistics on the NSG-UBR is similar to the workflow for NSG-BR. Refer to
BR VLAN Statistics (page 196) for workflow details and a sample output on an NSG-BR.

4.3.7 Sample BFD Peer Configuration

7750 SR Sample

1. Connect a router interface to the UBR underlay uplink and configure a BFD session.

A:SR>config>router>if# /configure router interface "ovs6-port2-vlan5"

A:SR>config>router>if# info
----------------------------------------------
address 12.12.12.1/24
port 1/1/25:5
bfd 500 receive 500 multiplier 3
no shutdown
----------------------------------------------
A:SR>config>router>if#

2. Create a static route for any network reachable from the UBR underlay uplink connection and enable BFD. In
this sample, the next hop is the UBR underlay uplink IP at 12.12.12.12. This creates a BFD session with the
specified next hop.

A:SR# configure router static-route-entry 1.1.1.4/32

A:SR>config>router>static-route-entry# info
----------------------------------------------
next-hop 12.12.12.12
bfd-enable
no shutdown
exit

Cisco Sample

bfd map ipv4 10.100.96.222/32 50.50.1.1/32 test

bfd-template multi-hop test
interval min-tx 51 min-rx 51 multiplier 3

ip route static bfd 10.100.96.222 50.50.1.1

interface GigabitEthernet0/0/1

4.3. NSG Underlay Border Router 175

 VNS User Guide, Release 5.3.2

ip address 50.50.1.1 255.255.255.0

negotiation auto

4.3.8 CLI Show Command Sample

BFD CLI commands

*A:vsc# tools <nsg-ip> command "ovs-appctl bfd/show"

---- 0x280000:12.12.12.1 ----
Forwarding: false
Multihop: false
Detect Multiplier: 3
Concatenated Path Down: false
TX Interval: Approx 1000ms
RX Interval: Approx 500ms
Detect Time: now +15483136ms
Next TX Time: now -435ms
Last TX Time: now +495ms
Last Flap Time: 2017-06-20 19:08:32.637
Link Mastership Status (configured): n/a
Link Mastership Status (current): n/a

Local Flags: none

Local Session State: down
Local Diagnostic: Control Detection Time Expired
Local Discriminator: 0xcdcb90d5
Local Minimum TX Interval: 1000ms
Local Minimum RX Interval: 500ms

Remote Flags: none

Remote Session State: down
Remote Diagnostic: No Diagnostic
Remote Discriminator: 0x0
Remote Minimum TX Interval: 0ms
Remote Minimum RX Interval: 1ms

The following command can be used to assess BFD status. The Local Diagnostic and Remote Diagnostic parameters
reflect the BFD status. When an egress filter is dropping BFD packets on the remote device, the Local Diagnostic
reads “Control Detection Time Expired”. When an ingress filter is dropping BFD packets on the remote device, the
Local Diagnostic parameter reads “Neighbor Signaled Session Down” and the Remote Diagnostic parameter reads
“Control Detection Time Expired”.

*A:vsc# ovs-appctl nuage/bfd-show port2.4

name: port2.4 gen-id: 33

uplink-type: BR
bfd-sessions: 1
collective-bfd-status: up

---- 0x280000:11.11.11.1 ----

Forwarding: true
Multihop: false
Detect Multiplier: 3
Concatenated Path Down: false
TX Interval: Approx 500ms

4.3. NSG Underlay Border Router 176

 VNS User Guide, Release 5.3.2

RX Interval: Approx 500ms

Detect Time: now -1048ms
Next TX Time: now -351ms
Last TX Time: now +39ms
Last Flap Time: 2017-08-24 22:34:35.843
Link Mastership Status (configured): n/a
Link Mastership Status (current): n/a

Local Flags: none

Local Session State: up
Local Diagnostic: No Diagnostic
Local Discriminator: 0xce2e0060
Local Minimum TX Interval: 500ms
Local Minimum RX Interval: 500ms

Remote Flags: ctl

Remote Session State: up
Remote Diagnostic: No Diagnostic
Remote Discriminator: 0x1
Remote Minimum TX Interval: 500ms
Remote Minimum RX Interval: 100ms

UBR routes CLI command

*A:vsc# show vswitch-controller ubr-routes svcId 240229042

===============================================================================
UBR Routes
===============================================================================
Route Type Datapath ID Public IP Private IP Owner
Underlay ID Public Port Uplink ID
-------------------------------------------------------------------------------
vxlan 45.165.60.209 10.60.5.2 10.60.5.2 BGP_VPN
6292 4789 23340
vxlan 45.165.60.209 10.6.6.2 10.6.6.2 BGP_VPN
7262 4789 18211
vxlan 45.165.60.209 10.6.5.2 10.6.5.2 BGP_VPN
7926 4789 18778
vxlan 45.165.60.209 10.60.6.1 10.60.6.1 BGP_VPN
62981 4789 6253
vxlan 83.154.158.2 10.50.5.2 10.50.5.2 BGP_VPN
6292 4789 35707

*A:vsc# show vswitch-controller gateway ubr-groups 71.87.23.202

===============================================================================
Gateway UBR Groups
===============================================================================
UBR Group ID UBR Group Name Priority One Way Delay
-------------------------------------------------------------------------------
f4d079ff-79f1-4776-b98e-92cb22a67d94 Ubr-Grp-1 100 50
-------------------------------------------------------------------------------
No. of UBR Groups: 1
===============================================================================

4.3. NSG Underlay Border Router 177

 VNS User Guide, Release 5.3.2

4.4 NSG Border Router

• Overview (page 179)

• Terminology (page 180)
– Permission Model (page 180)
• Controller Topology (page 181)
• NSG-BR Redundancy (page 181)
• Multi-tenancy (page 182)
– IPsec Considerations (page 182)
• BR Backbone (page 182)
• Tunnel Shaping (page 183)
• Underlay Reachability (page 183)
• NSG-BR Configuration (page 183)
– CSPRoot Workflow (page 183)
– Instantiate NSG-BR (page 184)
– Bootstrap the NSG-BR (page 184)
– Assign Permission (page 184)
– Create BR Connection (page 186)
– Configure BFD (page 187)
– Enterprise Admin Workflow (page 188)
– Link Domains (page 188)
– Configure Demarcation Service (page 189)
– Define End-to-End ACLs across Linked Domains (page 192)
• BR Backbone Configuration (page 194)
– Add Static Route in DC Domain (page 194)
– Configure Demarcation Service (page 194)
• Tunnel Shaping Configuration (page 194)
– Define Tunnel Shaping Policy (page 194)
– Define Rate Limiter(s) (page 194)
– Create Tunnel Shaper QoS Policy (page 195)
– Assign Policy to Uplink VLAN (page 195)
– Configuration Recommendations (page 196)
• Statistics (page 196)
• Sample CLI output for Tunnel Shapers (page 197)
• Sample CLI Output for BFD Sessions (page 199)

4.4. NSG Border Router 178

 VNS User Guide, Release 5.3.2

4.4.1 Overview

NSG Border Router (NSG-BR) feature enables a secure demarcation point between the WAN and the DC underlay
networks. It provides seamless connectivity between NSG and non IPsec-capable endpoints (e.g. VRS, 7750 SR) by
linking domains, while maintaining a unified policy.

Fig. 4.15: DC Connectivity to Secure Branch

NSG-BR was introduced in Release 4.0.R4 as a new gateway personality that runs on an hardware appliance or as
NSG-V.
Key functionality includes:
• Providing connectivity between a WAN and a DC underlay
• Linking a domain in the WAN (referred to as Branch domain) to a domain in the DC underlay (referred to as
DC domain)
• Support for multiple tenants
• IPsec to VxLAN and VxLAN to VxLAN translation
• NSG-BR to NSG per destination tunnel shaping
• Support for existing vPort functionality on NSG-BR
• Access resiliency for non-BR ports that are:
– L2 domain vPorts
– L3 domain vPorts with BGP
• End-to-end ACL across linked domains
• Service chaining (page 295) using advanced forwarding to a redirection target in the DC domain
• BR backbone (VxLAN) domain (page 182) for connecting branch domains in different regions using domain
linking.
The feature utilizes Domain Linking capability.

4.4. NSG Border Router 179

 VNS User Guide, Release 5.3.2

Fig. 4.16: Domain Linking with NSG-BR

Note: L2 domains are only supported on NSG-BR without domain linking.

4.4.2 Terminology

• BR-port (Border Router Port) is a new NSG access port type that connects the NSG-BR to the DC underlay.
The BR-port does not act as an uplink port. There may be one or more BR-ports, in addition to access ports and
uplink ports (for single or dual-uplinks).

Note: Currently, VPorts on NSG-BR in the DC domain are not supported.

• WAN Underlay refers to the Internet underlay or any other Underlay that the NSG-BR’s uplink is connected to.
It typically has NSG endpoints, with and without encryption.
• DC Underlay refers to the DC underlay or any other underlay that the VRS’s or non IPsec-enabled NSG’s are
connected to, via a port other than the NSG uplink.
• Demarcation Service is the next-hop to be used in advertisement of leaked routes. Specifically on the NSG-BR,
demarcation service refers to:
– Uplink Port/VLAN to link DC domain to Branch domain
– BR-port/VLAN to link Branch domain to DC domain
• Destination Domain is the domain receiving routes from another domain.
• Source Domain is the domain exporting its own routes to the destination domain.

Permission Model

Use permission may be assigned at gateway level or at access port or access port/VLAN level, and allows an enterprise
user:
• to attach VPorts, create domain link for domains in enterprise, and create demarcation service, but not create
VLANs or add BR-connections.
Extend permission may be assigned at access port or access port/VLAN level, and allows enterprise user:
• At access port level, to create VLANs, attach VPorts, add BR-connections, create domain links for domains in
enterprise, and create demarcation service.

4.4. NSG Border Router 180

 VNS User Guide, Release 5.3.2

• At access port/VLAN level, to attach VPorts, add BR-connections, create domain links for domains in enterprise,
and create demarcation service.

4.4.3 Controller Topology

A high-level controller topology is depicted in the figure below. The NSG and NSG-BR may share the same controller.

Fig. 4.17: NSG-BR Controller Topology

4.4.4 NSG-BR Redundancy

There are different redundancy scenarios:

• Uplink redundancy on the WAN underlay with a dual-uplink NSG-BR.

Fig. 4.18: NSG-BR Uplink Redundancy

• Node redundancy on the DC underlay using two NSG-BRs. In such a configuration, when there are multiple
next hops, routes in the destination domain will be ECMP routes.

Fig. 4.19: NSG-BR Node Redundancy

• Access resiliency (NSG-RG) is supported on the NSG-BR for:

4.4. NSG Border Router 181

 VNS User Guide, Release 5.3.2

– L2 domain VPorts
– L3 domain VPorts with BGP
However, RG ports cannot be BR-ports (i.e. cannot have a BR connection on such ports).
A CSPRoot can configure a pair of NSG-BRs to form a Redundant Group (RG) using the same workflow (page 245)
as that for the NSG personality type and non-BR ports.
In a multi-tenanted NSG-BR, an RG port or RG port/VLAN is assigned use permission for one or more enterprises.
If a RG port is deleted, then the permissions are also deleted.

4.4.5 Multi-tenancy

For multi-tenancy, a NSG-BR is owned and instantiated by CSProot. The CSProot may assign use or extend
permission to different organizations (enterprises) at the gateway level or access port or VLAN level as explained
below.
For a single-tenant NSG-BR, the CSPRoot may assign permission to the enterprise in which the demarcation services
will be created as follows:
• Use or Extend permission at the gateway level
• Use or Extend permission at the access port or access port/VLAN level
For a multi-tenant NSG-BR, the CSPRoot may assign:
• Use or Extend permission at the access port or access port/VLAN level
• Multiple Use permissions to an access port/VLAN on which a BR connection exists. Thus, a BR connection
can be shared across tenants.

Note: You must avoid overlapping IP addresses when linking two BR domains across or within enterprises.

IPsec Considerations

On a multi-tenant NSG-BR, traffic to other NSGs is encrypted using tenant specific keys that the destination NSG
belongs to. Refer to IPsec (page 409) chapter.

4.4.6 BR Backbone

Fig. 4.20: BR Backbone Topology

4.4. NSG Border Router 182

 VNS User Guide, Release 5.3.2

In this use case a central DC (VxLAN) domain serves as a backbone between different NSG-BRs, and the different
branch domains are also linked into the central DC domain. This provides end-to-end connectivity between branch
domain in one region to a branch domain in another domain, via the backbone domain using NSG-BRs.
BR backbone configuration workflow (page 194) is provided later in this chapter.

4.4.7 Tunnel Shaping

Fig. 4.21: Per Destination Tunnel Shaping

This capability allows the NSG-BR to shape traffic towards the WAN underlay for a destination NSG. The tunnel
shaping policy will be set on the uplink connection object of the destination NSG. One tunnel shaping policy (per
NSG uplink) is defined regardless of the number of NSG-BR uplinks.
When domain on destination NSG in the WAN (Branch domain) is linked to a domain in the DC underlay, tunnel shaper
for destination NSG is enabled on NSG-BR and traffic egressing on NSG-BR uplink and going towards destination
NSG is shaped as per shaping values defined in tunnel shaper.
Tunnel shaping configuration workflow (page 194) is provided later in this chapter.

4.4.8 Underlay Reachability

Bidirectional forwarding detection (BFD) is supported as an option for NSG-BR underlay reachability verification.
The BFD configuration workflow (page 187) includes configuration of the the BFD session attributes and the peer in
the underlay to which the session is established. The underlay connection is considered to be available for routing
only if the BFD session is up. The capability ensures that traffic is not forwarded via an underlay connection if it is
not available. If the BFD session fails, the routes through the BR connection are withdrawn and rerouted.

4.4.9 NSG-BR Configuration

CSPRoot Workflow

To deploy an NSG-BR for demarcation services, CSPRoot needs to:

• Instantiate the NSG-BR
• Assign Permissions
• Bootstrap the NSG-BR

4.4. NSG Border Router 183

 VNS User Guide, Release 5.3.2

• Create BR connection(s)

Note: NSG-BR may be activated before or after permissions are assigned.

Instantiate NSG-BR

This workflow assumes that the CSPRoot has already created a Gateway Template (page 38) with personality NSG-BR.
Only the CSPRoot can instantiate an NSG-BR.
1. As CSPRoot, select a Gateway Template with personality type NSG-BR and right-click to select instantiate
option. The CSPRoot can also instantiate using the same workflow (page 51) as described for Enterprise Ad-
ministrator.
2. The CSPRoot is required to assign an Installer as part of the NSG-BR instantiation. For a single tenant NSG-
BR, the installer may be an enterprise user or a CSP user. However, for the multi-tenant NSG-BR, the installer
must be a CSP user.

Bootstrap the NSG-BR

Single factor and two factor authentication are supported on the NSG-BR per the following permission model:
• CSP owns the NSG-BR and if there are no tenant permissions on gateway level, then CSPRoot can assign an
installer and complete bootstrap.
• A single tenant has gateway level permissions on the NSG-BR, then:
– CSPRoot can assign an installer and complete bootstrap, or
– Tenant can complete the bootstrap
Auto-Bootstrap of NSG-BR has the following permission model:
• CSP owns the NSG-BR and if there are no tenant permissions on gateway level, then CSPRoot can complete
auto-bootstrap.
• A single tenant has gateway level permissions on the NSG-BR, then only CSPRoot can complete the bootstrap
process

Note: For auto-bootstrap of a multi-tenant NSG-BR, the following must be observed:

• Between the generation of the auto bootstrap metadata (iso etc) and the successful completion of the bootstrap
process, no permissions on the gw level should be changed.
• Gateway level permissions can be changed post bootstrap.
• If gateway level permissions are changed pre bootstrap, new auto bootstrap metadata (iso) needs to be generated.

Assign Permission

For single tenant NSG-BR, the workflow is:

1. As CSPRoot, select the NSG-BR instance and navigate to the permission icon in the rightmost panel.

2. Select the at the bottom of the panel to select, in the popup, an Organization and Allowed to use the object
permission.

4.4. NSG Border Router 184

 VNS User Guide, Release 5.3.2

Fig. 4.22: Assign Permission at Gateway Level

For multi-tenant NSG-BR, the workflow is:

1. As CSPRoot, select the NSG-BR instance -> access port or port/VLAN, and navigate to the permission icon
in the rightmost panel.

2. Select the at the bottom of the panel to select, in the popup, an Organization and set the permission as
either Allowed to use the object or Allowed to extend the object.

Fig. 4.23: Assign Permission at Port Level

4.4. NSG Border Router 185

 VNS User Guide, Release 5.3.2

Fig. 4.24: Assign Permission at VLAN Level

The selected organization(s) can create the demarcation service before the activation of the NSG-BR. However, route
leaking will begin only after the NSG-BR is activated.

Attention:
• In a multi-tenant scenario, the CSPRoot can assign multipe use permissions on BR VLANs (i.e. VLANs
with BR connections). For such a scenario, the workflow requires that the BR connection (page 186) be
created before the permission is assigned.
• If a BR connection with multiple use permissions is deleted, then the permissions are also deleted.

Create BR Connection

Any Access port VLAN can be enabled as a BR-port by adding a BR-connection on that VLAN.
The BR connection also allows configuration of the IP connection attributes of the DC underlay facing interface.

Fig. 4.25: Create BR Connection

1. As CSPRoot, select the NSG-BR instance and from the NS Ports panel, select an access port.

4.4. NSG Border Router 186

 VNS User Guide, Release 5.3.2

2. In the VLANs panel, select a VLAN (if one exists) or create a VLAN.

3. In the right most panel, select BR connection icon . Select the at the bottom of the panel to create a
BR-connection.
4. In the New BR Connection popup, specify the Connection as Static IP.
5. Enter the IP address and Gateway address.

Note:
• The only way to delete a BR-port is to delete a BR connection.

If you select BFD for the Reachability Verification parameter, then you must:
• Configure a BFD session and assign it to the uplink connection as described below.
• Configure the BFD peer as shown in the sample configuration (page 175).

Configure BFD

1. Click on in BFD Session panel.

2. In the popup, configure the following:
• Destination IP Address: Specifies the underlay peer to which the session is established.
• Timer (ms): Specifies the time interval, in milliseconds, between BFD messages.
• Multiplier: Specifies how many consecutive failed BFD messages must occur before the BFD session is de-
clared down.
• Enable Multi-hop: Specifies whether the destination IP is a multi-hop path.
Use the this CLI command (page 199) to view information on the BFD session. The Local Session State specifies
whether the BFD session is up or down.

Note:

4.4. NSG Border Router 187

 VNS User Guide, Release 5.3.2

• You can configure only one BFD session per port VLAN.
• BFD configuration only supported at the instance level.
• You must delete the BFD Session object before changing the Reachability Verification parameter to something
other than BFD.

Enterprise Admin Workflow

As Enterprise administrator, in the Organization view, navigate to Networks -> Layer 3 Domains -> Linking.
Select an instantiated domain displayed in the the Layer 3 Domains panel, and configure as follows to link a source
domain to the selected destination domain:

Link Domains

The objective is to link the Branch domain to a DC domain and vice versa (since links are typically created for both
directions). By linking two domains, the routes from the source domain are leaked into the destination domain.

Fig. 4.26: Linking Branch and DC Domains with NSG-BR

Note:
• Cross tenant domains can only be linked by CSPRoot. An enterprise administrator can only link domains in
their own enterprise.
• For cross-tenant linking when Branch domain is in Organization 1 and DC domain is in Organization 2, then the
NSG-BR must have some object with use/extend permission to Organization 1.

The following steps describe how to create a link from a source domain to a destination domain, using the NSG-BR.
To create a bidirectional link between a Branch domain and a DC domain these steps have to be followed for both
directions:

1. In the Links panel, select icon to link a source domain to the selected destination domain.

4.4. NSG Border Router 188

 VNS User Guide, Release 5.3.2

Fig. 4.27: Domain Linking Tab

2. The following graphic shows a DC domain as the destination domain. Select Link Type as Border Router, and
select a Source domain. Select Create.

Fig. 4.28: Define New Domain Link

3. Repeat to link in the other direction.

Configure Demarcation Service

The demarcation service specifies the next hop for routes being leaked from a source domain into a destination domain.

4.4. NSG Border Router 189

 VNS User Guide, Release 5.3.2

The demarcation service configuration is described for the two domain linking scenarios:
• Branch to DC (using the BR connection as next hop)
• DC to Branch (using the NSG-BR uplinks as next hop)
Branch to DC: If the source domain is a branch domain, then endoints participating in the DC domain will see the
routes from the branch domain with next hop as the BR-port IP address.

Fig. 4.29: Configure Demarcation Service (Branch to DC)

1. Select the Branch domain to DC domain link and in the Demarcation Services panel, select icon to config-
ure the demarcation service.
2. Select the Type as BR Port.
3. Select the NSG-BR as the Gateway on which the demarcation service is being created .
4. Specify the Port and VLAN, which will correspond to the BR-port on the NSG-BR.
5. Once demarcation service is created, the Route Distinguisher (used for tagging these next hop routes in the
destination domain) and Priority are auto-assigned and displayed as shown below.

4.4. NSG Border Router 190

 VNS User Guide, Release 5.3.2

Fig. 4.30: Demarcation Service (Branch to DC)

DC to Branch: If the source domain is a DC domain, then endpoints in the branch domain will see the routes from
the DC domain with next hop as the datapath ID of the NSG-BR.

Fig. 4.31: Configure Demarcation Service (DC to Branch)

1. Select the DC domain to Branch domain link and in the Demarcation Services panel, select icon to config-
ure the demarcation service.
2. Select the Type as NSG BR.
3. Select the NSG-BR on which the demarcation service is being created as the Gateway.
4. Once demarcation service is created, he Route Distinguisher (used for tagging next hop of leaked routes in
destination domain) and Priority are are auto-assigned and displayed as shown below.

4.4. NSG Border Router 191

 VNS User Guide, Release 5.3.2

Fig. 4.32: Demarcation Service (DC to Branch)

Note:
• If a source/destination domain is linked using two NSG-BRs by defining a demarcation service on each, then it
is considered an active-active resilient configuration with ECMP.
• Redundancy of BR-ports is not supported on ports of the same NSG-BR.

Define End-to-End ACLs across Linked Domains

The Ingress and Egress security policy configuration for linked domains is identical to the workflow defined in the
Nuage VSP User Guide (Configuring Security and Forwarding Policies section).
The primary difference is that when defining security policy entries:
• ACL Type must be set to Linked Domain
• An Enterprise for the destination domain must be optionally selected (see Note below). If not specified, then
the same enterprise as source is assumed.
• For linked domains, only zone/subnet/policy group are allowed as
– Ingress ACL destination options
– Egress ACL origin options

Note:
• End-to-end ACLs for cross tenant domains can only be created by CSPRoot.

4.4. NSG Border Router 192

 VNS User Guide, Release 5.3.2

Fig. 4.33: Defining Security Policy Entry

Fig. 4.34: Sample End-to-end ACLs for Linked Domain

4.4. NSG Border Router 193

 VNS User Guide, Release 5.3.2

4.4.10 BR Backbone Configuration

Add Static Route in DC Domain

Fig. 4.35: Add Static Route of Type Overlay

1. For the central DC domain, select the Static Route icon from the top right configuration actions.
2. In the popup, specify the Network as 0.0.0.0/0, and select the route Type as Overlay, and Next Hop IP as the
address of a resolved vPort in the DC domain.

Configure Demarcation Service

Using the workflows defined earlier:

1. Link (page 188) branch domain and DC domain (in each direction), and
2. Configure demarcation service (page 189) from branch to DC and DC to branch.
End-to-end connectivity between branch domains is established once the steps are repeated for each branch domain.

4.4.11 Tunnel Shaping Configuration

The workflow below describes how the tunnel shaping policy is configured. However, tunnel shaping for a destina-
tion NSG gets enabled on NSG-BR only when both of the following conditions are met:
• Egress QoS (page 319) is enabled on NSG-BR uplink. The procedure for NSG-BR is identical to that for
gateway personality type NSG.
• Domain on destination NSG in branch domain is linked (page 188) to a domain in the DC underlay.

Define Tunnel Shaping Policy

Define Rate Limiter(s)

Rate limiters may be defined by CSPRoot, or at instance level by Enterprise administrator.

Define Rate Limiter(s) per the workflow described in the Egress QoS (page 319) chapter.

4.4. NSG Border Router 194

 VNS User Guide, Release 5.3.2

Create Tunnel Shaper QoS Policy

The tunnel shaper specifies the bandwidth requirement for destination NSG from NSG-BR. It is specified on the
destination NSG, and for any domain on the NSG that is linked to DC, the tunnel shaper will be applied on NSG-BR
uplink. As a result, traffic egressing from NSG-BR uplink towards the destination NSG will be subjected to shaper
values as specified in tunnel shaper.
Tunnel shaper policy may be created by the CSPRoot or, at instance level by enterprise administrator (description
below is for CSPRoot workflow) .

1. As CSPRoot, navigate to Infrastructure -> Network Service Gateways -> Egress QoS Policy -> Tunnel
Shaper QoS Policies.
2. Create a new Tunnel Shaper QoS Policy by following the same workflow as for creating Egress QoS Policy and
assigning Rate Limiter(s).

Fig. 4.36: Tunnel Shaper QoS Policy

Assign Policy to Uplink VLAN

Once an Tunnel Shaper QoS Policy is created, it must be applied to a port VLAN on destination NSG in order to
activate the shaping behavior. This can be done at the Gateway Template level by the CSPRoot or at the Gateway
instance level by the enterprise admin.
If the specific Gateway instance has inherited Tunnel Shaper QoS policies from the gateway template, the enterprise
administrator can override them by associating new tunnel shaper QoS policies at the instance level. Any changes at
instance level will persist, and thus any subsequent changes at template level will not be propogated to the instance.
1. Navigate to select a Network Services Gateway -> Port -> VLAN. An untagged VLAN is shown in the example
graphic, but it may be tagged as well.

2. On the far right select the QoS icon . The panel below the icon will either display a Tunnel Shaper QoS
Policy already assigned or will allow you to assign a policy.

4.4. NSG Border Router 195

 VNS User Guide, Release 5.3.2

Fig. 4.37: Assign Tunnel Shaper QoS Policy

Configuration Recommendations

On an NSG-BR, there can be multiple tunnel shapers enabled and one Egress QoS policy on uplink.
To achieve proper Committed Information Rate (CIR) for each tunnel, it is recommended that:
• Total Committed information rate (CIR) (i.e. Overall Rate in the tunnel shaper policy) across all tunnel shapers,
plus the total of all child queue CIRs in Egress QoS policy, should not exceed Peak Information Rate (PIR) of
Parent in Egress QoS policy.
• Peak Information Rate (PIR) of Parent in Egress QoS policy should be less than or equal to port bandwidth.
In other words, with reference to the figure below,

Q0-CIR (Tunnel 1) + Q0-CIR (Tunnel 2)+ ... + Q0-CIR (Tunnel n) + Q1-CIR + Q2-CIR + Q3-
˓→CIR + Q4-CIR <= Q0-PIR

Q0-PIR <= Port Bandwidth

Fig. 4.38: Tunnel Shaping CIR

4.4.12 Statistics

When BR connections are present, statistics are available at Access port/VLAN level.

4.4. NSG Border Router 196

 VNS User Guide, Release 5.3.2

To open the graphical display on the VSD UI, select the statistics button associated with the access port/VLAN on
which the BR connection is defined.
There are two views:
• Bytes view: Transmit (Tx) and Received (Rx) Bytes.
• Packets view: Transmit (Tx)/Received (Rx) packets count, packets dropped count, errored packets count dis-
played.

Fig. 4.39: BR VLAN Statistics

4.4.13 Sample CLI output for Tunnel Shapers

*A:vsc# show vswitch-controller domain-linking

===============================================================================
Domain Linking Information
===============================================================================
Src Domain Src Domain Name Dst Domain Dst Domain Name Link Type
SvcId SvcId
-------------------------------------------------------------------------------
1829935225 Trusted Domain1 284372794 Unrusted Domain1 br
1458084265 Trusted Domain2 36790322 Untrusted Domain2 br
284372794 Unrusted Domain1 1829935225 Trusted Domain1 br
36790322 Untrusted Domain2 1458084265 Trusted Domain2 br
-------------------------------------------------------------------------------
No. of domain links: 4
===============================================================================

To view enabled tunnel shaper on an NSGBR:

*A:vsc# show vswitch-controller gateway ports 119.10.51.60 destinations

===============================================================================
Gateway Uplink Destination Table
===============================================================================

Gateway IP : 119.10.51.60 Gateway Port Name : port1

Port VLAN ID : 0

4.4. NSG Border Router 197

 VNS User Guide, Release 5.3.2

Dest Datapath ID : 78.167.177.103 Dest Uplink Type : primary1

Dest Uplink IP : 10.15.33.254
Dest Qos UUID : 7129e25f-9c0c-4ea7-a3a4-b8cc05918398

Dest Datapath ID : 129.123.194.218 Dest Uplink Type : primary1

Dest Uplink IP : 10.15.2.254
Dest Qos UUID : e2ffec9b-8c17-4c73-8f0f-2eaa4fe7048a
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
No. of Entries: 1
===============================================================================

To see details of a tunnel shaper on the uplink of NSGBR:

*A:vsc# /show vswitch-controller gateway ports 119.10.51.60 destinations port-name

˓→"port1" dest-id 78.167.177.103

===============================================================================
Gateway Uplink Destination QoS Table
===============================================================================

Gateway IP : 119.10.51.60 Gateway Port Name : port1

Port VLAN ID : 0
Dest Datapath ID : 78.167.177.103 Dest Uplink Type : primary1

Egr QoS Parent Queue Information

Egress Shaping PIR(Mbps) : 1.199 Egress Shaping PBS(kb) : 1000
Egress Shaping CIR(Mbps) : 1.199
Egress Shaping Svc Classes : n/a

QoS Queue ID : 1
Egress Shaping PIR(Mbps) : 0.200 Egress Shaping PBS(kb) : 1000
Egress Shaping CIR(Mbps) : 0.200
Egress Shaping Svc Classes : h1 nc

QoS Queue ID : 2
Egress Shaping PIR(Mbps) : 0.100 Egress Shaping PBS(kb) : 1000
Egress Shaping CIR(Mbps) : 0.100
Egress Shaping Svc Classes : h2 ef

QoS Queue ID : 3
Egress Shaping PIR(Mbps) : 0.699 Egress Shaping PBS(kb) : 1000
Egress Shaping CIR(Mbps) : 0.699
Egress Shaping Svc Classes : af l1

QoS Queue ID : 4
Egress Shaping PIR(Mbps) : 0.200 Egress Shaping PBS(kb) : 1000
Egress Shaping CIR(Mbps) : 0.200
Egress Shaping Svc Classes : be l2
-------------------------------------------------------------------------------
No. of Entries: 1
===============================================================================

4.4. NSG Border Router 198

 VNS User Guide, Release 5.3.2

4.4.14 Sample CLI Output for BFD Sessions

*A:vsc# tools <nsg-ip> command "ovs-appctl bfd/show"

---- 0x280000:12.12.12.1 ----
Forwarding: false
Multihop: false
Detect Multiplier: 3
Concatenated Path Down: false
TX Interval: Approx 1000ms
RX Interval: Approx 500ms
Detect Time: now +15483136ms
Next TX Time: now -435ms
Last TX Time: now +495ms
Last Flap Time: 2017-06-20 19:08:32.637
Link Mastership Status (configured): n/a
Link Mastership Status (current): n/a

Local Flags: none

Local Session State: down
Local Diagnostic: Control Detection Time Expired
Local Discriminator: 0xcdcb90d5
Local Minimum TX Interval: 1000ms
Local Minimum RX Interval: 500ms

Remote Flags: none

Remote Session State: down
Remote Diagnostic: No Diagnostic
Remote Discriminator: 0x0
Remote Minimum TX Interval: 0ms
Remote Minimum RX Interval: 1ms

The following command can be used to assess BFD status. The Local Diagnostic and Remote Diagnostic parameters
reflect the BFD status. When an egress filter is dropping BFD packets on the remote device, the Local Diagnostic
reads “Control Detection Time Expired”. When an ingress filter is dropping BFD packets on the remote device, the
Local Diagnostic parameter reads “Neighbor Signaled Session Down” and the Remote Diagnostic parameter reads
“Control Detection Time Expired”.

*A:vsc# ovs-appctl nuage/bfd-show port2.4

name: port2.4 gen-id: 33

uplink-type: BR
bfd-sessions: 1
collective-bfd-status: up

---- 0x280000:11.11.11.1 ----

Forwarding: true
Multihop: false
Detect Multiplier: 3
Concatenated Path Down: false
TX Interval: Approx 500ms
RX Interval: Approx 500ms
Detect Time: now -1048ms
Next TX Time: now -351ms
Last TX Time: now +39ms
Last Flap Time: 2017-08-24 22:34:35.843
Link Mastership Status (configured): n/a
Link Mastership Status (current): n/a

4.4. NSG Border Router 199

 VNS User Guide, Release 5.3.2

Local Flags: none

Local Session State: up
Local Diagnostic: No Diagnostic
Local Discriminator: 0xce2e0060
Local Minimum TX Interval: 500ms
Local Minimum RX Interval: 500ms

Remote Flags: ctl

Remote Session State: up
Remote Diagnostic: No Diagnostic
Remote Discriminator: 0x1
Remote Minimum TX Interval: 500ms
Remote Minimum RX Interval: 100ms

4.4. NSG Border Router 200

 VNS User Guide, Release 5.3.2

4.5 Network Uplinks (Wired and LTE)

• Overview (page 202)

• LTE Uplink (page 202)
• Use Case Assumptions (page 202)
• Use Cases (page 203)
– Heterogeneous transport network - Tunnel (VxLAN/IPSec) (page 203)
– Heterogeneous transport network - no Tunnel (page 203)
• LTE Staging and Operations Requirements (page 204)
– Verifying port mode (page 204)
– Changing SIMs for integrated LTE devices (page 206)
• Provisioning Overview (Wired and LTE Uplinks) (page 206)
– LTE Specific Configuration (page 206)
• Uplink Role (page 207)
• Configuring Controller(s) (page 208)
• Forwarding Behavior (page 210)
– Uplink Preference (page 210)
• Network Acceleration (page 212)
– Network Acceleration Configuration (page 212)
– Network Acceleration Verification (page 213)

* Verify network acceleration has been enabled (page 213)

* Verify the time network acceleration was enabled (page 213)
* Verify hugepages allocation (page 214)
• NSG Controller Selection (page 214)
• Controller Resiliency (page 215)
– Scenario 1: NSG uplink(s) not behind a NAT device (page 215)

* Failure Handling (page 215)

– Scenario 2: NSG uplink is behind NAT device (page 216)

* Failure Handling (page 216)

• IPv6 uplinks (page 217)
– Feature support (page 217)
• Uplink Statistics (page 218)
– NSG Uplink Statistics Example (page 219)
– NSG-UBR Uplink Statistics Examples (page 221)
– LTE Uplink Statistics, Status and Dongle Information (page 222)

4.5. Network Uplinks (Wired and LTE) 201

 VNS User Guide, Release 5.3.2

* LTE Dongle Information (page 225)

• CLI Samples (page 226)
– Network Acceleration CLI Samples (page 228)

4.5.1 Overview

In order to meet increasing bandwidth demand, introduce resiliency or segregate network traffic, the NSG supports
multiple uplinks using wired connections and wireless LTE uplinks. Support for network uplinks enables the follow-
ing:
• Resiliency for the data and control paths.
• A mechanism to classify traffic (if desired). For example, high priority traffic can be routed over an expensive
secure link and lower priority traffic over a cheaper unprotected link.

4.5.2 LTE Uplink

LTE uplink provides additional flexibility for WAN connectivity and enhances service assurance, reliability, and time
to new service delivery in locations where wired connections are not easily procured. LTE uplink capabilities and
limitations are the following:
• LTE connection is supported only on NSG-C and NSG-E.
• The LTE uplink is supported using a USB dongle or integrated LTE hardware. The dongles can be modem-based
or ethernet-based.
• IPv6 is not currently supported with LTE uplinks (integrated and USB-based).
• Dual LTE uplinks are currently not supported. LTE uplinks can be USB-based or integrated, but not both.
• USB-based LTE uplinks are not currently supported on NSG platforms E206/306.
• The LTE uplink may be used as a primary link or as a transport of last resort with selective traffic offload.
Advanced ACLs or AAR allow traffic redirection to the expensive LTE uplink either in active-active or auxiliary
mode.
• Statistics and Status (page 222) of the LTE dongle is monitored by VSD and the statistics collector.

Note: Refer to the Nuage VNS Release Notes for the currently supported use case scenarios and interaction with
features, such as BGP, PAT, QoS, AAR etc..

4.5.3 Use Case Assumptions

1. An Enterprise Domain may contain NSGs (including personalities NSG-BR, NSG-UBR) with either single or
dual (wired) uplinks. Regular NSGs may also have one LTE uplink (as a single uplink) or with a wired uplink
(in a dual uplink scenario).
2. The wired dual uplinks may be connected to the same transport network (IPVPN or Internet) or to two hetero-
geneous networks (one IPVPN and other Internet). In the latter case, the transport may be provided by the same
or different providers. It is assumed that routing is possible between distinct transport networks (including
LTE). Deployment consideration must be given to ensure that this condition is met to avoid possible traffic
black-holing during certain transport network failures.

4.5. Network Uplinks (Wired and LTE) 202

 VNS User Guide, Release 5.3.2

3. Each network uplink (wired or LTE) is tagged with a preference, namely; primary or secondary (refer to Uplink
Role (page 207)). Link selection preference is an overlay only concept.
4. A VSC may connect to different NSGs via uplinks of different priority. For example, the same VSC may connect
to NSG 1 via the NSG’s primary uplink and to NSG 2 via the NSG’s secondary uplink.
5. The transport networks which an NSG’s uplinks are part of may have NAT devices in their paths. (refer to the
NAT-T feature of VNS).
6. When both uplinks are tagged as primary, the NSG will equally load-balance traffic across both uplinks. Such a
configuration will not form LAG-group with the two uplinks as members.
7. In a dual uplink scenario, each uplink must be on networks with different IP addresses.

Note: Multi-tenancy on the NSG is not supported. In other words, a NSG may belong to a single Enterprise with
multiple subnets associated with one or more VRFs.

4.5.4 Use Cases

Heterogeneous transport network - Tunnel (VxLAN/IPSec)

In this use case, it is expected that each uplink is connecting the NSG to distinct transport networks. Based on customer
specified policies, the service traffic traverses the heterogeneous networks.

Fig. 4.40: Heterogeneous Transport Network - Tunnel

In scenarios with NAT vs Non-NAT, the forwarding behaviour is the same, but the control-plane behaviour will be
different.

Heterogeneous transport network - no Tunnel

This use case is an application of interworking with the Port Address Translation feature. For subnets that have PAT
enabled, there are two cases:
• Address Pool Not Configured: Once the correct uplink interface is determined as part of the forwarding, the
source IP of the outgoing packet will be the selected uplink’s WAN IP address.

4.5. Network Uplinks (Wired and LTE) 203

 VNS User Guide, Release 5.3.2

Fig. 4.41: Heterogeneous Transport Network - No Tunnel, No Address Pool

• Address Pool Configured: Since the link selection preference is an overlay only concept, the underlay (in the
case of PAT) will be selected based on kernel routing look-up for the destination address. Generally this will be
the default routing. It would most probably mean that Port 1 will always be used as its default route would be
preferred.

Fig. 4.42: Heterogeneous Transport Network - No Tunnel, Address Pool

Note: In the case of heterogeneous transport networks, it is recommended that customers use portable public IP
address pools and that the transport providers are provisioned for the proper connectivity required for routing under
the various link failure scenarios.

4.5.5 LTE Staging and Operations Requirements

Attention: The set of supported LTE devices is specified in the Hardware Compatibility List provided in the
VNS Release Notes.

• LTE service/hardware procurement/shipping is the responsibility of the service provider/enterprise.

• It is recommended that the LTE equipment be initialized and tested per the carrier specific requirements before
being shipped to the installer.
• All dongles are expected to be in unlocked condition.

Verifying port mode

For all the Modem based dongles, the modem should be configured as the first port mode. To verify dongle information
and/or modify the LTE dongle’s port mode settings, proceed as follows:
• Plug the Dongle into a PC/laptop
• Connect to the serial port of the dongle (e.g using putty in Windows or minicom in Linux). Identify the serial
port (e.g. COM4) using the device manager.
• The following commands can be issued in the connected console:
– Determine dongle model: ATI

4.5. Network Uplinks (Wired and LTE) 204

 VNS User Guide, Release 5.3.2

Example output:

Manufacturer: huawei
Model: E3372
Revision: 21.286.03.00.00
IMEI: 866133026907917
+GCAP: +CGSM,+DS,+ES

– Determine dongle version: AT^VERSION?

Example output:

^VERSION:BDT:Apr 25 2014, 19:05:48

^VERSION:EXTS:21.286.03.00.00
^VERSION:INTS:
^VERSION:EXTD:UTPS23.015.05.01.03_MAC23.015.05.00.03
^VERSION:INTD:
^VERSION:EXTH:CL1E3372SM Ver.A
^VERSION:INTH:
^VERSION:EXTU:E3372
^VERSION:INTU:
^VERSION:CFG:1004
^VERSION:PRL:
^VERSION:OEM:
^VERSION:INI:

– Verify dongle’s port mode: AT^GETPORTMODE.

Modem should be listed first ahead of PCUI. If modem port is not configured as first port, then the dongle’s port
mode order must be modified (page 205).
Example output 1

^GETPORTMODE: TYPE: WCDMA: huawei,pcui:1,3g_modem:2,ncm:3,mass:4,mass_two:5,

Note: Output may differ for different dongle models.

Example output 2

^GETPORTMODE:TYPE:WCDMA:Qualcomm,MDM:0,PCUI:1,DIAG:2,CDROM:5,SD:6

• To modify dongle’s portmode:

Step 1 Get the current port configuration AT^SETPORT?
Example output

^SETPORT:A1,A2;12,1,16

Step 2 Change the port configuration (replace A1,A2 with FF) and reset the device. Note that map-
ping might differ between dongles.
Example output

AT^SETPORT="FF;1,12,16"
AT^RESET

Step 3 Reconnect to the serial port and verify that port mode is modified AT^GETPORTMODE

4.5. Network Uplinks (Wired and LTE) 205

 VNS User Guide, Release 5.3.2

Example output 1

^GETPORTMODE: TYPE: WCDMA: huawei,3g_modem:1,pcui:2,ncm:3,

Example output 2

^GETPORTMODE: TYPE: WCDMA: huawei,,3g_modem:0,pcui:1,ncm:2,3g_diag:

˓→3,4g_diag:4,mass:5,mass_two:6

Changing SIMs for integrated LTE devices

Integrated LTE devices have two SIM slots, but currently only one SIM per device is supported. Insert only a single
SIM into any of the two slots (slot 1 or slot 2). The system will automatically detect the slot in which the SIM
is inserted and use that to bring up LTE connection. If no SIM is detected in any slot, the integrated LTE uplink
configuration is aborted and logged on the NSG.

Note: When adding, removing, or swapping a SIM in an integrated LTE device, the NSG must always be powered
down.

4.5.6 Provisioning Overview (Wired and LTE Uplinks)

Uplink(s) may be configured at either template or instance level as follows:

1. Configure Ports of type Network (page 40) for wired and LTE uplink connections whereby the physical interface
is specified.
2. Add VLAN(s) (page 42).
3. Define Uplink Connection (page 43) whereby the connection mode is identified as Dynamic, Static, PPPoE or
LTE link.
4. Assign Infrastructure VSC profile (page 43).

LTE Specific Configuration

Note: It is recommended that the LTE device (USB or integrated) get a publicly routable IP (static or dynamic).
However, if the provider of the LTE device has assigned a private IP address, then NAT-T (page 143) must be enabled
(Full Cone). In addition, provider should allow inbound connection to the opened ports, i.e. CGNAT done by carrier
should also be full cone.

• LTE uplink is configured by setting the uplink connection mode parameter as LTE. Options for wired connec-
tions are Dynamic, Static, and PPPoE.
• For modem-based dongles, custom properties, such as APN information, authentication type (PAP/CHAP-
default), authentication credentials, and SIM PIN, are configured via LTE Uplink Connection Custom Properties
in VSD. Note that only a subset of custom properties are supported for integrated LTE devices. Refer to LTE
Mode Parameters (page 57) for specifics.
• The LTE interface is only supported on network port and an untagged VLAN. Any other configuration will
cause the bootstrapping to fail.
• The physical interface name for LTE uplink must be configured as lteX, where X is any integer.

4.5. Network Uplinks (Wired and LTE) 206

 VNS User Guide, Release 5.3.2

• An auxiliary mode is provided to allow an operator to configure the LTE uplink as a circuit of last resort.
In the auxiliary mode, the LTE interface is activated only when the wired connection is down for more than 30 seconds.
Traffic will be impacted until the LTE link becomes operational, after which traffic will be routed via the LTE interface.
Once the wired connection is back up, the auxiliary port is brought down immediately and traffic is switched to the
wired uplink.

4.5.7 Uplink Role

Uplinks can have primary or secondary role. Primary refers to the preferred uplink. A pair of uplinks can be
Primary/Secondary or both primary or both secondary. If role is not specified, then the default is assumed to be
primary on Port 1 and secondary on Port 2. Role is specified as part of the Connection Type (page 59) by the Enterprise
Admininstrator.
It is recommended, but not mandatory, that on the Dual-Uplink deployments, both uplinks have a different role (one
Primary and one Secondary).

Note: In the scenario when one uplink is an LTE connection and is configured as an auxiliary uplink, then it may
have the same uplink role as the port it will protect to guarantee that the traffic, the wired uplink was carrying, is
successfully redirected through the LTE Uplink. However, based on how the advanced ACLs have been defined, an
operator may prefer having the Role of the Auxiliary LTE Uplink to differ from the one of the wired uplink.

The role is sent to the NSG as part of the infrastructure configuration procedures either via bootstrapping or through a
subsequent configuration reload request.
The roles translate to internally maintained Route Distinguishers (RDs) that get created (as part of the Domain class)
to uniquely identify the routes advertised by the VSCs. The RDs are not user configurable. Each uplink has internal
RDs, one for each role (as applicable) as follows:
Uplink Tag RD Preference
Port 1 Primary1 RDPrimary1 200
Port 1 Secondary1 RDSecondary1 100
Port 2 Primary2 RDPrimary2 200
Port 2 Secondary2 RDSecondary2 100
The RDs can be viewed by navigating to the Networks tab and selecting the Domain. Clicking on the instance icon
( ), in the middle panel, displays the RDs in the far right panel. You may need to scroll the right panel to view the
RD information.

4.5. Network Uplinks (Wired and LTE) 207

 VNS User Guide, Release 5.3.2

Fig. 4.43: Uplink Route Distinguishers

At any point, based on NSG’s network uplink profile(s), two of the four RDs will be advertised by VSC. The RD
information is passed from the VSD to the VSC at the vPort resolution.
Examples (assuming NSG is not behind a NAT):
1. If the port 1 uplink role is Primary, and port 2 uplink Secondary for NSG which has subnet 10.0.0.0/24, then
VSC will advertise 10.0.0.0/24 via uplink 1 as (RDPrimary1, NH port 1) and via uplink 2 as ( RDSecondary2,
NH port 2).
2. If the port 1 uplink role is Primary, and port 2 uplink Primary for NSG1 which has subnet 10.0.0.0/24, then VSC
will advertise 10.0.0.0/24 via uplink 1 as (RDPrimary1, NH port 1) and via uplink 2 as ( RDPrimary2, NH port
2). When a remote NSG receives these routes, based on RDs being both primary, it knows it has to ECMP to
uplink1 and uplink2 of NSG1.

4.5.8 Configuring Controller(s)

Each network port/VLAN must connect to at least one controller or to a pair of controllers. The IP address of the
controller(s) is specified in the Infrastructure VSC profile, and there can be up to two controller IP addresses in each
infrastructure VSC profile. Refer to Infrastructure VSC Profile (page 37) for details of the configuration workflow.

4.5. Network Uplinks (Wired and LTE) 208

 VNS User Guide, Release 5.3.2

The VSC profile is attached by CSPRoot to Network port VLAN Templates (Refer to Assign VSC Profile (page 43)).
The CSPRoot may also assign VSC Profile at NSG instance level if it is not inherited from the template.
A network port/VLAN is associated with an infrastructure VSC profile with one or two VSCs. The controllers for an
NSG with dual uplinks may be configured as follows:
• A pair of distinct controllers using two VSC profiles with one controller in each. Port 1 VLAN is associated
with one profile and Port 2 VLAN with the other.

Fig. 4.44: Dual Uplinks with two Controllers

• Two pairs of distinct controllers (or a variation thereof)using two VSC profiles whereby each network port
VLAN is associated with one VSC profile.

Fig. 4.45: Dual Uplinks with more than two Controllers

Note: Dual uplinks cannot connect to a single controller or to a set of controllers from the same VSC profile.

NSG establishes OF-TLS connection to each of the controllers. As part of the OpenFlow initialization, the NSG
notifies the controller of its uplink information, specifically number of uplinks, uplink IP address / priority, and NAT
flag. If any of the NSG’s uplink is behind a Full Cone NAT, then a DTLS session is also established with every

4.5. Network Uplinks (Wired and LTE) 209

 VNS User Guide, Release 5.3.2

controller that has an established OF-TLS connection (refer to NAT-T (page 130)). An NSG determines that one or
both of its uplinks are behind a NAT device based on a flag received as part of the bootstrap process.
Using an internal controller selection (page 214) procedure, the NSG designates one of the controllers as Primary and
the others are designated Secondary. The terms Primary and Secondary are documentation constructs for describing
the behaviour of the controllers for route advertisements and for programming network reachability information to the
NSGs.
All configured controllers advertise the reachability information pertaining to NSG subnets (in a domain) to other
remote NSGs in the same domain. However, the Primary Controller advertises with a higher local preference, and
the Secondary controller advertises with a lower local preference. The primary controller is the only controller that
programs all the network reachability information to the local NSG via the established OF-TLS connection.
The resiliency of the controller depends on whether the NSG uplink(s) is behind a NAT device or not (refer to Con-
troller Resiliency (page 215)).
By default, the VSC prefers the outband management interface for XMPP communication with the VSD. If the route to
the VSD is not present in the outband management routing table, the VSC attempts to establish an XMPP connection
using the inband management interface. If the route to the VSD is not present in the inband management routing table,
the VSC attempts to establish an XMPP connection by checking the router “Base” routing table. The user can change
this default behaviour using the following command:
[no] route-preference primary [inband | outband] secondary [inband | outband |
none]
Refer to the VSP User Guide for more information on the route-preference command.
The default control interface of the VSC only supported untagged traffic. The user can create a new VLAN-tagged
control interface by creating a new interface and disabling the default control interface using the following command:
configure router interface <name> create configure router interface control
shutdown configure router interface control no address

4.5.9 Forwarding Behavior

The default domain forwarding behaviour for overlay traffic destined to remote VTEPs (egressing the network
ports) is defined in the table. Assuming there are two operational links, the default behaviour if one link fails is
summarized below:
Local Dual Uplink State Remote Next Hop Forwarding Decision
Primary Up RDPrimary received Uplink = Primary, NH = Primary
Primary Up RDSecondary received Uplink = Primary, NH = Secondary
Primary Down RDPrimary received Uplink = Secondary, NH = Primary
Primary Down RDSecondary received Uplink = Secondary, NH = Secondary
If remote NSG has both links marked as primary, then per flow based ECMP (not weighted) routing will be performed.

Uplink Preference

The default forwarding behavior can be overridden by selecting or changing the local or remote uplink preference
using the steps as described below.
In a dual uplink scenario, you can specify either a Local Uplink Preference or a Remote Uplink Preference to
determine the forwarding behavior of traffic on the source or destination NSG. Only one of these preference parameters
can be configured; one of them must remain as Default.

4.5. Network Uplinks (Wired and LTE) 210

 VNS User Guide, Release 5.3.2

Remote uplink preference configuration is intended for the use case where an NSG-UBR is forwarding traffic to a
destination NSG. The local uplink preference cannot be specified for an NSG-UBR. For traffic forwarded from an
NSG to another NSG, the remote uplink preference should not be configured.
The forwarding behavior of each advanced ACL option is described in the table below. For
each option, the remote next hop behavior is that Primary is always preferred over Secondary:
ACL Local Uplink Selection
Default n/a
Primary, Secondary Prefer Primary uplink if operational, else Secondary, else drop
Secondary, Primary Prefer Secondary uplink if operational, else Primary, else drop
Primary Primary if operational, else drop
Secondary Secondary if operational, else drop

Note: For symmetric traffic flow, apply end-to-end ACLs on all NSGs.

Note: See SLA-Aware Path Selection (page 449) for information about forwarding behavior when the SLA-Aware
parameter is enabled.

Note: The following procedure describes only the parameters pertaining to L7 ACLs as they pertain to dual uplink
and AAR path selection. See the “Security and Forwarding Policies” chapter in the VSP User Guide for complete
information about configuring L7 ACLs.

1. In the Organization view, select Networks -> Layer 3 Domains ( ) -> Policies.

2. Select the Ingress Forwarding Policies icon icon to display a list of existing Forwarding Policies or allow
the creation of a new policy using .

Fig. 4.46: New Forwarding Policy

Forwarding policies can be created by both the CSPRoot and Organization Admin user and can be created as part of
domain or zone template or after instantiation.

3. Once the policy is created, click in the Security Policy Entries panel to display the New Ingress Forwarding

4.5. Network Uplinks (Wired and LTE) 211

 VNS User Guide, Release 5.3.2

Policy Entry popup.

4. Click on the Actions tab.
5. Configure the Local Uplink Preference, Remote Uplink Preference, or SLA-Aware parameters, as required.
6. Configure all other parameters, as required, and click Create.

4.5.10 Network Acceleration

Network acceleration can be enabled on the NSG by offloading encryption and decryption functionality. This optimiza-
tion is achieved using the Data Plane Development Kit (DPDK). The DPDK feature set supports branch deployments,
VPort handoff at hub sites, BR port handoff at hub sites, and UBR disjoint underlay connections.
There are two options for network acceleration on the NSG:
• None: This option disables network acceleration. This is the default option.
• Performance: This option enables separate threads for packet encryption and decryption, and separate threads
for KNI input and output. This option is suitable for the highest throughput systems.
When the Network Acceleration parameter is configured, a Reload Configuration is required before the changes can
take effect. This triggers an NSG system reboot. Network acceleration changes take effect once the NSG successfully
reboots.
Consider the following when enabling or disabling network acceleration on an NSG:
• Network acceleration is supported on all physical NSG platforms.
• Network acceleration is functionally supported on the NSG-V.
• Network acceleration is not supported for LTE or PPPoE uplinks.
• Network acceleration is not supported on NSGs configured with static IP. If the feature is enabled on an NSG
configured with static IP, the NSG may lose uplink connectivity.
• Network acceleration is supported for overlay-only data paths. Network acceleration should not be enabled in
branches with underlay traffic, such as IKE tunnels, PAT-to-underlay, route to underlay, or Internet breakout
traffic. Network underlay traffic throughput is not improved by enabling network acceleration.

Network Acceleration Configuration

Note: Modifying the Network Acceleration parameter on an NSG may cause service interruption.

Note: Network acceleration cannot be enabled on an NSG pre-bootstrap. The NSG must first be upgraded and
bootstrapped to the latest version before the feature can be enabled.

Perform the following steps to configure network acceleration on an NSG.

1. From the Infrastructure tab, click Network Services Gateways.
2. Right-click on an NSG and choose Edit.
3. Configure the Network Acceleration parameter and click Update.

4.5. Network Uplinks (Wired and LTE) 212

 VNS User Guide, Release 5.3.2

4. Click on Reload Configuration in the Bootstrap tab of the NSG properties form.
5. If you are disabling network acceleration, a warning dialog will ask you to confirm the action before rebooting
the NSG.
6. If required, use the commands listed in the following section to verify that network acceleration has been suc-
cessfully enabled.

Network Acceleration Verification

CLI commands can be executed to verify whether network acceleration has been successfully enabled on the NSG.
Refer to Network Acceleration CLI Samples (page 228) for sample outputs for commands related to network acceler-
ation.

Verify network acceleration has been enabled

To confirm that network acceleration has been successfully enabled, execute the -bash-4.2$ grep
"PLATFORM=" /etc/default/openvswitch command and verify that PLATFORM=dpdk is displayed.

-bash-4.2$ grep "PLATFORM=" /etc/default/openvswitch

PLATFORM=dpdk

Verify the time network acceleration was enabled

To confirm when network acceleration was enabled, execute the -bash-4.2$ systemctl status vrs_dpdk
command and verify that Active: active (running) is displayed with the date and time that network accel-
eration was enabled.

4.5. Network Uplinks (Wired and LTE) 213

 VNS User Guide, Release 5.3.2

-bash-4.2$ systemctl status vrs_dpdk

vrs_dpdk.service - LSB: Open vSwitch DPDK
Loaded: loaded (/etc/rc.d/init.d/vrs_dpdk)
Active: active (running) since Tue 2018-02-27 20:37:26 UTC; 4h 23min ago
Process: 4126 ExecStart=/etc/rc.d/init.d/vrs_dpdk start (code=exited, status=0/
˓→SUCCESS)

CGroup: /system.slice/vrs_dpdk.service
\u251c\u25004174 /bin/sh /bin/uplink_monitor.sh
\u251c\u25004578 ovsdb-server: monitoring pid 4579 (healthy)
\u251c\u25004579 ovsdb-server /home/root/nsg-agent/ovs-db/conf.db -
˓→vconsole:...

\u251c\u25004619 /usr/bin/python /sbin/nuage-SysMon -vany:console:emer -

˓→vany...

\u251c\u25004620 /usr/bin/python /sbin/nuage-SysMon -vany:console:emer -

˓→vany...

\u251c\u25004672 vrs-dpdk: monitoring pid 4673 (healthy)

\u251c\u25004673 vrs-dpdk unix:/var/run/openvswitch/db.sock -vconsole:emer
˓→-...

\u251c\u25007515 sleep 1
\u2514\u25007516 n/a

Verify hugepages allocation

To confirm that hugepage sizes have been correctly allocated for 1G pages, execute the -bash-4.2$
/proc/cmdline command and verify that default_hugepagesz=1G hugepagesz=1G hugepages=4
is displayed.

-bash-4.2$ /proc/cmdline
initrd=initrd0.img root=live:UUID=4829dced-492b-298f-29fe-299a-3920-9382938f903
˓→rootfstype=ext3 ro rd.live.image quiet panic=2 net.ifnames=0 console=ttyS0,115200n8

˓→rd.luks=0 rd.md=0 rd.dm=0 isolcpus=4-15 nohz=on nohz_full=4-15 rcu_nocbs=4-15

˓→default_hugepagesz=1G hugepagesz=1G hugepages=4 BOOT_IMAGE=vmlinuz0

4.5.11 NSG Controller Selection

The VSC profile is downloaded to the NSG as part of the Infrastructure configuration. The NSG internal algorithm
for selecting evaluates the controllers using an ordered list to avoid any conflicts in VSC IP address. The ordered list
is created as follows:
• Port1 controllers ordered 1st
• First VSC ordered prior to Second VSC
The ordered list and the resulting selection for some sample configuration scenarios is shown below:

4.5. Network Uplinks (Wired and LTE) 214

 VNS User Guide, Release 5.3.2

Fig. 4.47: VSC Selection Examples

Note: Once the NSG selects a controller as Primary, it notifies the corresponding controller about being designated
Primary as part of the OF-TLS handshake. If the Primary controller fails, then another controller is declared Primary
and the behaviour is not revertive. In other words, the new Primary continues to function as Primary going forward,
and when the previous controller gets restored, it becomes Secondary.

4.5.12 Controller Resiliency

In the case of an NSG with Dual Uplinks, a resilient controller configuration depends on whether the NSG uplink(s)
is behind a NAT device or not.
The scenarios are described using an illustrative example that assumes the following:
• Two VSC profiles, each with a distinct controller, VSC1 and VSC2.
• Each profile is associated with port1 and port2 respectively.
• port1 connects to VSC1 and port2 connects to VSC2, and the NSG designates VSC1 to be Primary controller.
• Uplink attached to port1 is tagged as “primary” and uplink on port 2 is tagged as “secondary”

Scenario 1: NSG uplink(s) not behind a NAT device

VSC1 and VSC2 advertise routes with Next Hop (NH) port1:pref1 and also advertises routes with NH port2:pref2
(lower preference). Note that VSC1 (selected Primary by the NSG) will advertise both port1 and port2 subnet routes
with higher preference while VSC2 will advertise same route with lower preference.

Failure Handling

If VSC1 fails, since VSC2 advertises NH for port1, port1 NH can still be used (as it is assumed that port1 is reachable
via network that port2 is connected to). Similarly if VSC2 fails, then too port2 NH is reachable.
Any link failure / IP address changes to port (e.g. due to DHCP) cause routes to be withdrawn / updated.
Example: VSC1 advertises routes with NH port1:pref1 and NH port2:pref2. If uplink on port2 fails or port2 IP address
changes, then VSC1 advertisement will either withdraw or change NH depending on former or latter event.

4.5. Network Uplinks (Wired and LTE) 215

 VNS User Guide, Release 5.3.2

If OF session to VSC2 fails, there is no change on VSC1 route advertisement behaviour. Of course this may lead to
corner cases where traffic may be black holed.
In the event of link failure, the two controllers back each other up and provide resiliency for the control and data path.

Scenario 2: NSG uplink is behind NAT device

If the NSG uplink is behind a Full Cone NAT, it sets up DTLS sessions with VSC1 and VSC2. The DTLS session is
used to relay NAT traversal mapping route to the VSC using the NSG’s unique identifier (DatapathID) and the uplink
RD as the key.
NAT mapping routes provide the information as to how to reach an NSG behind a NAT device on a specific uplink,
using a specific tunnel type (IPSEC or VXLAN).
All local NAT mappings are redistributed to BGP which advertises them to other IBGP peers in a proprietary EVPN
route.
All subnets belonging to an NSG are advertised to the rest of the network with the next hop set to the NSG DatapathID
along with an indicator that it is a virtual next hop. In routing, when a virtual next hop is encountered, there is a second
lookup of NAT mapping table to determine reachability to that NSG. Primary VSC will advertise subnet routes with a
higher local preference than the secondary VSC(s).
Example:
• VSC1 will generate a NAT mapping route (internal IP:port to NATed IP:port) learnt via DTLS session for port1,
and include the RD belonging to port1 (e.g. primary)
• VSC2 will also generate a NAT mapping route (internal IP:port to NATed IP:port) learnt via DTLS session for
port2, and include the RD belonging to port2 (e.g. secondary)
• Both VSCs will receive each others NAT mapping and also advertise to other VSCs.
• VSC1 will advertise subnet routes with Virtual NH set to the NSG’s datapath ID, and will also indicate that it
is a virtual next hop. VSC2 will do the same with a lower local preference. VSC1 will also program the local
NSG with the subnet routes.
• Receiving (remote) NSGs will have the subnet information with virtual NH and the NAT mapping routes. For
routing, a lookup for the destination subnet will result in a virtual next hop route. The NSG will then perform a
second lookup of the NAT mapping table for determining the higher priority next hop and route accordingly.

Failure Handling

If either VSC1 or VSC2 fails, or OF/DTLS session dies, or path between NSG and VSC fail, then the subnet and NAT
mappings routes are withdrawn and as such remote NSG will not have an entry for routing for the port on which the
VSC has failed. In other words, the VSCs do not offer any redundancy for routing.
If Controller resiliency is required, then it is recommended that two VSC profiles be defined, each with distinct VSCs.
Associate Profile 1 with port1 and Profile 2 with port2. This means the NSG will have 4 VSCs (two on each uplink).
One of the 4 VSCs will be declared as Primary for subnet route advertisements, and all others will be secondary. In
such a configuration, the NAT mapping and subnet routes for each port will be advertised by two distinct VSCs, and
so even if one VSC fails, the other controller will provide redundancy.

Note: Failure scenarios can include physical port failure, network path failure, or controller failure. In order to
maintain the same expected behaviour under all failure conditions, fate sharing between data and control-plane is
assumed. Thus, detection of OF-TLS session failure is indicative of link or network path failure.

4.5. Network Uplinks (Wired and LTE) 216

 VNS User Guide, Release 5.3.2

4.5.13 IPv6 uplinks

IPv6 can be enabled on static or dynamic uplinks by configuring the Address Family parameter on the uplink con-
nection configuration form. Refer to Define Uplink Connections (page 54) for more information about configuring
uplinks.
Consider the following before configuring a network uplink for IPv6:
• If a VSC infrastructure profile with IPv6 is assigned to an uplink VLAN, IPv6 must be enabled on the network
uplink.
• IPv6 must be enabled at the domain level before IPv6 can be selected for an uplink connection. Refer to the
VSP User Guide for more information about configuring L2 or L3 domains.
• NAT probes are enabled by default on network ports, but NAT traversal is not supported for IPv6. The probes
must be disabled before configuring an IPv6 uplink connection. Disable the NAT Probes Enabled parameter
on the NSG port configuration form before creating an IPv6 uplink connection. Refer to Configuring NAT-T
settings at the NSG Port level (page 143) for more information.

Feature support

IPv6 support is planned to be introduced over multiple releases of the VNS solution. The following table lists the
current level of IPv6 support for VNS features.

Feature IPv6 Support

Installation and Upgrade
VSP upgrade Not supported
VNS active/standby Not supported
NSG factory software upgrade Not supported
IPv6-supported NSG factory software Not supported
NSG Models
NSG-C Supported
NSG-E Supported
NSG-X Supported
NSG-V Supported
NSG-AWS Not supported
VSD Operations
Infrastructure access profiles Not supported
Infrastructure gateway profiles Supported
Infrastructure VSC profiles Supported
NSG Operations
Auto-bootstrapping Supported
One/Two factor bootstrapping Supported
CLI-based bootstrapping Supported
Pre-bootstrapping functionality Not supported
NSG log collextion Supported
Proxy reachability Supported
Controllerless operation Supported
CLI-based NSG operations Supported
SSH for NSG Not supported
NSG upgrade profiles Not supported
NSG patching Not supported
Auto-bootstrapping Supported
Continued on next page

4.5. Network Uplinks (Wired and LTE) 217

 VNS User Guide, Release 5.3.2

Table 4.1 – continued from previous page

Network Infrastructure
NAT Traversal Not supported
Secondary IP on NSG Not supported
NSG-UBR Supported
NSG-BR Not supported
Network Services
DHCP on bridge ports Supported
Access resiliency Not supported
PAT to underlay Not supported
PAT to overlay Not supported
Bidirectional NAT Not supported
NSG dual stack Not supported
Dual WAN uplinks (IPv6+IPv6) Supported
Static uplinks Supported
DHCP uplinks Supported
LTE uplinks Not supported
PPPoE uplinks Not supported
IPv6 Vports Not supported
L2 domains Not supported
L3 domains Supported
Service chaining Not supported
DNA subnets Not supported
QoS Not supported
Routing Protocols
BGP Not supported
OSPF Not supported
Encryption
Group-Key IPsec Supported
IKE IPsec Not supported
Branch Services
AAR Not supported
AAR visualization Not supported
VNFs Not supported
WiFi on NSG Not supported

4.5.14 Uplink Statistics

Uplink statistics collection is enabled by default when an NSG is bootstrapped.

The uplink statistics may be:
• Viewed graphically via the Statistics display window on the VSD UI,
• Accessed via VSD ReST API.
Uplink statistics are reported at:
• Network port/VLAN level
• Network port level (aggregated across VLANs)
• Connection level (for LTE uplinks)
The collection interval is set to 60 seconds and is not configurable.

4.5. Network Uplinks (Wired and LTE) 218

 VNS User Guide, Release 5.3.2

The following statistics are displayed graphically in the statistics window, and also available via ReST API:
• Bytes: total number of bytes egressing out of a queue
• Packet Count: total number of packets egressing out of a queue

To open the graphical display on the VSD UI, select the statistics button associated with either:
• network port, or
• network port/VLAN
There are two views:
• Bytes view: Transmit (Tx) and Received (Rx) Bytes.
• Packets view: Transmit (Tx)/Received (Rx) packets count, packets dropped count, errored packets count dis-
played.

NSG Uplink Statistics Example

1. Bytes view for NSG Control VLAN

4.5. Network Uplinks (Wired and LTE) 219

 VNS User Guide, Release 5.3.2

Fig. 4.48: Bytes view for NSG Control VLAN

2. Packets view for NSG Control VLAN

4.5. Network Uplinks (Wired and LTE) 220

 VNS User Guide, Release 5.3.2

Fig. 4.49: Packets view for NSG Control VLAN

NSG-UBR Uplink Statistics Examples

3. Bytes and Packets views for an NSG-UBR Control VLAN

4.5. Network Uplinks (Wired and LTE) 221

 VNS User Guide, Release 5.3.2

Fig. 4.50: NSG-UBR Control VLAN Statistics

4. Bytes and Packets views for an NSG-UBR Underlay VLAN

Fig. 4.51: NSG-UBR Underlay VLAN Statistics

LTE Uplink Statistics, Status and Dongle Information

To open the graphical display on the VSD UI, select the statistics button associated with the LTE connection uplink
(via Infrastructure -> NSG -> Port -> VLAN -> ).

Note: The VSD UI displays statistics data over a 2-hour window for a given start time. In the UI, the start time may
be varied but the window is fixed at 2 hours. However, via ReST, statistics for a longer duration may be retrieved.

Select one of the tabs, States, Ratios, Connection Types:

1. States: Displays the LTE service state and readiness to route traffic. The following states are supported:
• Inactive: Dongle is plugged in but processes for the specific modem (i.e. wvdial for modem based, and nmcli d
connect for Ethernet based) failed to start. Thus, the dongle is not connected to the network.
• Active: Dongle is plugged in processes for the specific modem successfully started. Thus, the dongle should be
connected to the network.

4.5. Network Uplinks (Wired and LTE) 222

 VNS User Guide, Release 5.3.2

• Ready: This state applies when the dongle is configured in auxiliary mode. The dongle is plugged in, but no
services running on it, and it is not connected to the network.
• Available: Reflects the initial state of the dongle.

Fig. 4.52: LTE Service States

Note: The states displayed on the VSD reflect the status of the modem processes on the NSG. In addition, the dongle
itself will have LED status indicators as defined by the provider.

2. Ratios: Displays the signal strength.

4.5. Network Uplinks (Wired and LTE) 223

 VNS User Guide, Release 5.3.2

Fig. 4.53: LTE Signal Strength

3. Connection Types: Displays the types of LTE connection established.

4.5. Network Uplinks (Wired and LTE) 224

 VNS User Guide, Release 5.3.2

Fig. 4.54: LTE Connection Types

LTE Dongle Information

To view information about the LTE dongle, select the information icon associated with the LTE connection uplink
(via Infrastructure -> NSG -> Port -> VLAN -> ). Note that some information in the example graphics below is
intentionally obfuscated.

Fig. 4.55: LTE Dongle Information Example 1

4.5. Network Uplinks (Wired and LTE) 225

 VNS User Guide, Release 5.3.2

Fig. 4.56: LTE Dongle Information Example 2

4.5.15 CLI Samples

Look for Dual uplink Mappings in the output below:

*A:vsc# show vswitch-controller vswitches detail

===============================================================================
Virtual Switch Table
===============================================================================
vswitch-instance : va-10.15.3.254/5
Personality : NSG
Uptime : 0d 00:05:02 VM Count : 0
Num of hostIf : 4 Num of bridgeIf : 4
Num of multiVMs : 0
OF version : 1 OF nego. version : 1
OF Conn. port : 6633
Cntrl. role : primary Cntrl. Conn. type : tls
Cntrl. auth type : tls Cntrl. crl lookup : false
Cntrl. Conn. mode : secure
Cntrl. Conn. state : ready
Cntrl. client verification : false
Cntrl. client IP verification : false
Gateway Hold Time(sec) : 15 Gateway Echo Time(sec) : 5
Gateway Topic : nuage_gateway_id_194.211.53.254
Gateway Retry/Audit Time : 644
XMPP error code : 0
XMPP error text : (Not Specified)
JSON Conn. State : Up
Static Peer : False
Datapath-Id : va-194.211.53.254

-------------------------------------------------------------------------------
Dual Uplink Table
-------------------------------------------------------------------------------
IP Addr Preference Behind
NAT
-------------------------------------------------------------------------------

4.5. Network Uplinks (Wired and LTE) 226

 VNS User Guide, Release 5.3.2

10.15.3.254 primary no
10.18.3.254 secondary no
-------------------------------------------------------------------------------
No. of Dual Uplink Mappings: 2
-------------------------------------------------------------------------------

-------------------------------------------------------------------------------
No. virtual switches: 1
===============================================================================

NAT mappings indicating route to be PRIMARY/SECONDARY VXLAN/IPSEC

*A:vsc# show vswitch-controller nsg nat-traversal-mapping svc-id 20001

===============================================================================
NSG NAT Traversal Mappings
===============================================================================

------------------------------------------------------------------------------
Legend:
Flag : P -> Primary, S -> Secondary, V -> VXLAN, I -> IPSEC
------------------------------------------------------------------------------
Flags Service Datapath Id Public Private Site Owner
Id Ip Address Ip Address Id
Port
------------------------------------------------------------------------------
PV 20001 12.240.216.51 10.15.33.254 10.15.33.254 0 BGP_VPN
4789
PV 188.102.166.2* 10.15.1.254 10.15.1.254 0 NVC
4789
SV 208.166.92.194 10.18.3.254 10.18.3.254 0 NVC
4789
PI 12.240.216.51 10.15.33.254 10.15.33.254 0 BGP_VPN
4500
PI 188.102.166.2* 10.15.1.254 10.15.1.254 0 NVC
4500
PI 208.166.92.194 10.15.3.254 10.15.3.254 0 NVC
4500
------------------------------------------------------------------------------
No. of NAT Traversal Mappings: 6
------------------------------------------------------------------------------

ACL forwarding order for dual uplink.

*A:vsc# show vswitch-controller vports type host detail acl ingress-adv-fwd

===============================================================================
Virtual Port Ingress Advanced Fwding ACL Table
===============================================================================
VP Name : va-10.15.1.254/1/3
VLAN ID : 3
ACL UUID : 00000000-0000-0000-0000-000000000000
Priority : 0
Proto : 0
Src IP : 0.0.0.0/0
Min. Src Port : 0
Max. Src Port : 0
Min. Dest Port : 0

4.5. Network Uplinks (Wired and LTE) 227

 VNS User Guide, Release 5.3.2

Max. Dest Port : 0

Match DSCP : 0xff
Flow Log : Disabled
Stats Log : Disabled
FC Override : -
Redirect Target : -
Action : Fwd
Policy Grp Tag : 0:0
Uplink Fwd Order : none

-------------------------------------------------------------------------------
Total No. of Ingress Adv Fwd ACL's: 1
===============================================================================

*A:vsc# show vswitch-controller vports type host

==============================================================================
Virtual Port Table
==============================================================================
VP Name G/W PortName VLAN ID VPRN EVPN
VP IP Address WiFi Vport
VP IPv6 Address
------------------------------------------------------------------------------
va-105.137.212.225/1/3 port3 3 1449530380 2003665765
20.0.0.4/24 False
n/a
va-105.137.212.225/1/4 port3 4 1449530380 2003665765
20.0.0.5/24 False
n/a
va-105.137.212.225/1/7 port3 7 1449530380 289558567
20.0.1.4/24 False
n/a
va-105.137.212.225/1/8 port4 16 1449530380 289558567
20.0.1.5/24 False
n/a
va-70.109.71.78/1/3 port4 3 1449530380 237822901
20.0.2.4/24 False
n/a
va-70.109.71.78/1/4 port4 4 1449530380 237822901
20.0.2.5/24 False
n/a
va-70.109.71.78/1/7 port4 6 1449530380 2133782633
20.0.3.4/24 False
n/a
va-70.109.71.78/1/8 port4 7 1449530380 2133782633
20.0.3.5/24 False
n/a
------------------------------------------------------------------------------
No. of virtual ports: 8
==============================================================================

Network Acceleration CLI Samples

This section lists CLI output samples for commands that are useful for monitoring and debugging network acceleration.
The following command can be used to view a list of available commands.

4.5. Network Uplinks (Wired and LTE) 228

 VNS User Guide, Release 5.3.2

-bash-4.2$ ovs-appctl -t vrs-dpdk list-commands

The available commands are:
autoattach/show-isid [bridge]
autoattach/statistics [bridge]
autoattach/status [bridge]
bfd/set-forwarding [interface] normal|false|true
bfd/show [interface]
bond/disable-slave port slave
bond/enable-slave port slave
bond/hash mac [vlan] [basis]
bond/list
bond/migrate port hash slave
bond/set-active-slave port slave
bond/show [port]
bridge/dump-flows bridge
bridge/reconnect [bridge]
cfm/set-fault [interface] normal|false|true
cfm/show [interface]
coverage/show
dpctl/add-dp dp [iface...]
dpctl/add-flow [dp] flow actions
dpctl/add-if dp iface...
dpctl/ct-bkts [dp] [gt=N]
dpctl/ct-stats-show [dp] [zone=N] [verbose]
dpctl/del-dp dp
dpctl/del-flow [dp] flow
dpctl/del-flows [dp]
dpctl/del-if dp iface...
dpctl/dump-conntrack [dp] [zone=N]
dpctl/dump-dps
dpctl/dump-flows [dp] [filter=..] [type=..]
dpctl/flush-conntrack [dp] [zone=N]
dpctl/get-flow [dp] ufid
dpctl/list-commands
dpctl/mod-flow [dp] flow actions
dpctl/normalize-actions actions
dpctl/parse-actions actions
dpctl/set-if dp iface...
dpctl/show [dp...]
dpif-netdev/ethtool [dp]
dpif-netdev/global-stats [dp]
dpif-netdev/pmd-rxq-show [dp]
dpif-netdev/pmd-stats-clear [dp]
dpif-netdev/pmd-stats-show [dp]
dpif-netdev/pmd-stats-tab [dp]
dpif-netdev/vrs-stats [dp]
dpif/dump-dps
dpif/dump-flows [-m] [--names | --no-nmaes] bridge
dpif/port-cfg [dp]
dpif/port-stats
dpif/ports
dpif/set-dp-features bridge
dpif/show
dpif/show-dp-features bridge
exit [--cleanup]
fdb/flush [bridge]
fdb/show bridge
ipsec/perf-test-mode [on/off]

4.5. Network Uplinks (Wired and LTE) 229

 VNS User Guide, Release 5.3.2

ipsec/show-in-sas summary|stats|detail
ipsec/show-out-sas summary|stats|detail
ipsec/show-tun summary|stats|detail
ipsec/stats display|reset
lacp/show [port]
list-commands
mdb/flush [bridge]
mdb/show bridge
memory/show
netdev-dpdk/detach pci address of device
netdev-dpdk/mbuf-info
netdev-dpdk/set-admin-state [netdev] up|down
ofproto/list
ofproto/trace {[dp_name] odp_flow | bridge br_flow} [OPTIONS...] [-
˓→generate|packet]

ofproto/trace-packet-out [-consistent] {[dp_name] odp_flow | bridge br_flow}

˓→[OPTIONS...] [-generate|packet] actions

ovs/route/add ip_addr/prefix_len out_br_name [gw] [pkt_mark=mark]

ovs/route/del ip_addr/prefix_len [pkt_mark=mark]
ovs/route/lookup ip_addr [pkt_mark=mark]
ovs/route/show
qos/show interface
qos/show-types interface
revalidator/purge
revalidator/wait
rstp/show [bridge]
rstp/tcn [bridge]
stp/show [bridge]
stp/tcn [bridge]
tnl/arp/flush
tnl/arp/set BRIDGE IP MAC
tnl/arp/show
tnl/egress_port_range min max
tnl/neigh/flush
tnl/neigh/set BRIDGE IP MAC
tnl/neigh/show
tnl/ports/show -v
upcall/disable-megaflows
upcall/disable-ufid
upcall/enable-megaflows
upcall/enable-ufid
upcall/set-flow-limit flow-limit-number
upcall/show
version
vlog/close
vlog/disable-rate-limit [module]...
vlog/enable-rate-limit [module]...
vlog/list
vlog/list-pattern
vlog/reopen
vlog/set {spec | PATTERN:destination:pattern}

The following command can be used to view a list of all traffic flows on the alubr0 and netbr0 bridges.
-bash-4.2$ ovs-appctl -t vrs-dpdk dpif/dump-flows alubr0
tunnel(tun_id=0x472aa6,src=112.137.36.114,dst=40.61.143.51,tos=0xc0,ttl=62,flags(-df-
˓→csum+key)),skb_mark(0),recirc_id(0),in_port(12),eth(src=00:0b:ab:59:29:26,dst=00:00:

˓→0a:0a:32:01),eth_type(0x0800),ipv4(src=50.94.0.54,dst=50.50.0.

˓→26,proto=17,ttl=63,frag=no),udp(src=13001,dst=13501), packets:389265, bytes:

˓→150279856, used:0.000s, actions:set(eth(src=68:54:ed:00:b1:fb,dst=00:00:c6:0c:ea:

˓→60)),set(ipv4(src=50.94.0.54,dst=50.50.0.26,ttl=62)),set(skb_mark(0xc0)),11
4.5. Network Uplinks (Wired and LTE) 230
 VNS User Guide, Release 5.3.2

The following command can be used to view a list of tunnels and ports with network acceleration enabled.
-bash-4.2$ ovs-appctl -t vrs-dpdk dpif/show
netdev@dpdk-netdev: hit:3700596560 missed:97971
alubr0:
alubr0 65534/2: (tap)
ti292022711dab3d0c 70/12: (vxlan: df_default=false, dst_port=5770,
˓→key=4663974, local_ip=40.61.143.51, remote_ip=113.34.32.41)

ti292022714e1f 52/12: (vxlan: df_default=false, dst_port=5770, key=16777213,

˓→local_ip=40.61.143.51, remote_ip=113.34.32.41)

ti33f787911dab3d0c 29/12: (vxlan: df_default=false, dst_port=5770,

˓→key=4663974, local_ip=40.61.143.51, remote_ip=145.135.247.51)

The following command can be used to view a list of IPsec in SAs and related statistics.
-bash-4.2$ ovs-appctl -t vrs-dpdk ipsec/show-in-sas stats
INDX SPI SRC DST DPORT PKTS AUTH_F CRYPTO_F OLD_SEQ SAME_SEQ
˓→PAD_ERR

7840 6efbd275 40.61.143.51 174.100.188.137 4500 0 0 0 0 0

˓→ 0
7704 bd4df8f5 40.61.143.51 173.82.234.246 4500 0 0 0 0 0
˓→ 0
7910 feb0214a 40.61.143.51 137.165.165.52 4500 0 0 0 0 0
˓→ 0
7653 89c0e00e 40.61.143.51 174.100.188.137 4500 0 0 0 0 0
˓→ 0
7612 1c862733 40.61.143.51 87.247.198.55 4500 0 0 0 0 0
˓→ 0
7796 d3cb31c6 40.61.143.51 87.247.198.55 4500 0 0 0 0 0
˓→ 0

The following command can be used to view a list of IPsec out of SAs and related statistics.
-bash-4.2$ ovs-appctl -t vrs-dpdk ipsec/show-out-sas stats
INDX SPI SRC DST DPORT PKTS AUTH_F CRYPTO_F OLD_SEQ SAME_SEQ
˓→PAD_ERR

7840 6efbd275 40.61.143.51 174.100.188.137 4500 0 0 0 0 0

˓→ 0
7704 bd4df8f5 40.61.143.51 173.82.234.246 4500 0 0 0 0 0
˓→ 0
7910 feb0214a 40.61.143.51 137.165.165.52 4500 0 0 0 0 0
˓→ 0
7653 89c0e00e 40.61.143.51 174.100.188.137 4500 0 0 0 0 0
˓→ 0
7612 1c862733 40.61.143.51 87.247.198.55 4500 0 0 0 0 0
˓→ 0
7796 d3cb31c6 40.61.143.51 87.247.198.55 4500 0 0 0 0 0
˓→ 0

The following command can be used to view a list of ethtool commands.

-bash-4.2$ ovs-appctl -t vrs-dpdk dpif-netdev/ethtool -h
dpif-netdev/ethtool [-p <port-name>] [-h] [-k] [-s] [-q] [-v] [-x] [-c]
-p: specify port for which the remainder of the command should apply to.
if this option is not specified, the command applies to all ports
-c: clear statistics
-h: help

4.5. Network Uplinks (Wired and LTE) 231

 VNS User Guide, Release 5.3.2

-k: show features

-s: statistics
-q: queue statistics
-x: extended statistics
-v: verbose

The following command can be used to view network acceleration IPsec statistics.

-bash-4.2$ ovs-appctl -t vrs-dpdk ipsec/stats

tun_add_dup_ignored : 240 tun_del_hash_failures : 0
˓→ tun_find_success : 1247
tun_add_alloc_failures : 0 tun_del_lookup_failures : 0
˓→ tun_lkup_failures : 0
tun_add_hash_failures : 0 tun_del_success : 64
tun_add_success : 248

outsa_add_alloc_failures : 0 outsa_del_success :
˓→ 7040 outsa_lkup_active_success : 850815464
outsa_add_tun_lkup_failures : 0 outsa_del_spi_lkup_failures : 0
˓→ outsa_lkup_backup_success : 0
outsa_add_success : 8072 outsa_del_tun_lkup_failures : 16
˓→ outsa_lkup_failures : 0

insa_add_alloc_failures : 0 insa_del_hash_failures : 0
˓→ insa_lkup_hash_failure : 0
insa_add_hash_failures : 0 insa_del_lkup_failures : 0
˓→ insa_lkup_success : 442501775
insa_add_dup_ignored : 480 insa_del_success :
˓→ 7360
insa_add_success : 7544

The following command can be used to view network acceleration global statistics and errors.

-bash-4.2$ ovs-appctl -t vrs-dpdk dpif-netdev/global-stats

rx_access: 1404373159 tx_access: 0
rx_uplink: 870881936 tx_uplink: 0
rx_ring_net: 1098774555 tx_ring: 1098774555
rx_other: 11882 tx_pkts: 2274976453
push_tnl_clone: 0 rule_miss: 205
pop_tnl_clone: 0 odp_del: 206448
recirc_clone: 0 nsh_del: 0
clone_pkt: 345 no_pmd: 6
bad_pkt: 0 upcall_err: 83681
upcall_start: 94998 upcall_end: 11316
encrypt_start: 1098774559 encrypt_end: 1098774553
decrypt_start: 566481305 decrypt_end: 566481305
rx_all: 3374041528 execute: 4550342064
meter_drop: 0 push_tnl_err: 0
local_upcall_err: 3 userspace_err: 0
userspace_clone: 0 userspace_fwd: 3
rx_vlan_drops: 0 rx_policer_drops: 0
emc_insert: 1287 emc_clear: 813
emc_overlap: 461 emc_insert_fail: 0
pkt_allocs: 3374041877 pkt_frees: 3374041345

The following command can be used to view PMD statistics and view per-core utilization for network acceleration.

4.5. Network Uplinks (Wired and LTE) 232

 VNS User Guide, Release 5.3.2

-bash-4.2$ ovs-appctl -t vrs-dpdk dpif-netdev/pmd-stats-show

pmd thread numa_id 0 core_id 4:
emc hits:100974235
megaflow hits:7603
avg. subtable lookups per hit:1.00
miss:406
lost:0
idle cycles:42375373035352 (99.17%)
processing cycles:355278620460 (0.83%)
avg cycles per packet: 423150.15 (42730651655812/100982244)
avg processing cycles per packet: 3518.23 (355278620460/100982244)
pmd thread numa_id 0 core_id 5:
emc hits:100969870
megaflow hits:7778
avg. subtable lookups per hit:1.00
miss:9811
lost:0
idle cycles:42392146327268 (99.21%)
processing cycles:338471996912 (0.79%)
avg cycles per packet: 423127.97 (42730618324180/100987459)
avg processing cycles per packet: 3351.62 (338471996912/100987459)
pmd thread numa_id 0 core_id 6:
emc hits:75722118
megaflow hits:7773
avg. subtable lookups per hit:1.00
miss:371
lost:0
idle cycles:42469875190328 (99.39%)
processing cycles:260785228436 (0.61%)
avg cycles per packet: 564248.15 (42730660418764/75730262)
avg processing cycles per packet: 3443.61 (260785228436/75730262)
pmd thread numa_id 0 core_id 7:
emc hits:1558435709
megaflow hits:40936
avg. subtable lookups per hit:1.00
miss:7956204
lost:0
idle cycles:41363329166916 (96.80%)
processing cycles:1367329508696 (3.20%)
avg cycles per packet: 27278.96 (42730658675612/1566432849)
avg processing cycles per packet: 872.89 (1367329508696/1566432849)
pmd thread numa_id 0 core_id 8:
emc hits:92286897
megaflow hits:6602
avg. subtable lookups per hit:1.00
miss:25707
lost:79
idle cycles:42532442893900 (99.54%)
processing cycles:198205814548 (0.46%)
avg cycles per packet: 462857.63 (42730648708448/92319206)
avg processing cycles per packet: 2146.96 (198205814548/92319206)
pmd thread numa_id 0 core_id 9:
emc hits:204804034
megaflow hits:27142
avg. subtable lookups per hit:1.01
miss:6633
lost:0
idle cycles:42340444015620 (99.09%)

4.5. Network Uplinks (Wired and LTE) 233

 VNS User Guide, Release 5.3.2

processing cycles:390200759884 (0.91%)

avg cycles per packet: 208607.21 (42730644775504/204837809)
avg processing cycles per packet: 1904.93 (390200759884/204837809)
pmd thread numa_id 0 core_id 10:
emc hits:1904820151
megaflow hits:2864
avg. subtable lookups per hit:1.11
miss:48283
lost:129
idle cycles:38007948118304 (88.95%)
processing cycles:4722714544020 (11.05%)
avg cycles per packet: 22432.31 (42730662662324/1904871298)
avg processing cycles per packet: 2479.28 (4722714544020/1904871298)
pmd thread numa_id 0 core_id 11:
emc hits:513549157
megaflow hits:8586
avg. subtable lookups per hit:1.01
miss:1091
lost:0
idle cycles:41850863255176 (97.94%)
processing cycles:879805048564 (2.06%)
avg cycles per packet: 83205.01 (42730668303740/513558834)
avg processing cycles per packet: 1713.15 (879805048564/513558834)
pmd thread numa_id 0 core_id 12:
emc hits:58905071
megaflow hits:5382
avg. subtable lookups per hit:1.00
miss:971
lost:0
idle cycles:42524615304728 (99.52%)
processing cycles:206050066788 (0.48%)
avg cycles per packet: 725337.51 (42730665371516/58911424)
avg processing cycles per packet: 3497.62 (206050066788/58911424)
pmd thread numa_id 0 core_id 13:
emc hits:819885717
megaflow hits:9982
avg. subtable lookups per hit:1.00
miss:1515
lost:0
idle cycles:40531431638064 (94.85%)
processing cycles:2199235092068 (5.15%)
avg cycles per packet: 52117.10 (42730666730132/819897214)
avg processing cycles per packet: 2682.33 (2199235092068/819897214)
pmd thread numa_id 0 core_id 14:
emc hits:33655411
megaflow hits:2405
avg. subtable lookups per hit:1.00
miss:247
lost:0
idle cycles:42610916183672 (99.72%)
processing cycles:119723590380 (0.28%)
avg cycles per packet: 1269551.36 (42730639774052/33658063)
avg processing cycles per packet: 3557.06 (119723590380/33658063)
pmd thread numa_id 0 core_id 15:
emc hits:75724710
megaflow hits:5618
avg. subtable lookups per hit:1.00
miss:664

4.5. Network Uplinks (Wired and LTE) 234

 VNS User Guide, Release 5.3.2

lost:0
idle cycles:42482768164080 (99.42%)
processing cycles:247887136940 (0.58%)
avg cycles per packet: 564242.65 (42730655301020/75730992)
avg processing cycles per packet: 3273.26 (247887136940/75730992)
main thread:
emc hits:0
megaflow hits:0
avg. subtable lookups per hit:0.00
miss:0
lost:0
idle cycles:1685390056 (100.00%)
processing cycles:0 (0.00%)

The following command can be used to view per-port statistics. Valid port options are <uplink>-net, dpdkr1,
and <access>.

Port port1-net

RxPkts: 1077322583 TxPkts: 1817188658 Rx 1-64: 39

RxBytes: 528802188177 TxBytes: 907746475707 Rx < 128: 11128
RxErrs: 0 TxErrs: 0 Rx < 256:
˓→659592378

RxDrops: 0 TxDrops: 0 Rx < 512: 167

RxMcast: 0 TxMcast: 0 Rx < 1024:
˓→252831403

RxBcast: 9 TxBcast: 20 Rx < 1523: N/A

Rx > 1522: N/A

4.5. Network Uplinks (Wired and LTE) 235

 CHAPTER

FIVE

NETWORK SERVICES

• DHCP on Bridge Ports (page 237)

• Access Resiliency (page 242)
• Port Address Translation (page 255)
• PAT to Overlay (page 277)
• Bi-Directional NAT (page 287)
• Service Chaining in VNS (page 295)
• DNA Subnets (page 303)
• QoS (page 307)

236
 VNS User Guide, Release 5.3.2

5.1 DHCP on Bridge Ports

• Overview (page 237)

– DHCP Decline Handling (page 238)
• Provisioning Workflow (page 238)
– Define DHCP Pool (page 238)
– Modifying DHCP Address Ranges (page 240)

* DHCP Lease Time Preference (page 241)

– DHCP Support (page 241)
• CLI Show Command Sample (page 241)

5.1.1 Overview

In a VNS deployment, it is expected that multiple hosts are connected to the NSG via one or more host ports and/or
bridge ports as depicted in the figure Host Connectivity to NSG via Host and Bridge Port Types (page 237). Whenever
a virtual port is instantiated in a domain, a unique IP address may be assigned based on the subnet to which it belongs.
The DHCP on Bridge Ports feature refers to NSG’s ability to service requests from multiple unique hosts connected
behind the same vPort-type bridge and allocating a unique IP address to all such host requests.
The NSG serves as the DHCP server for the subnet. The DHCP pool is created in the VSD as part of the subnet
definition. In addition, the feature also allows for static MAC-to-IP address bindings to be defined in the VSD in order
to assign specific IP addresses for a set of MAC addresses.

Fig. 5.1: Host Connectivity to NSG via Host and Bridge Port Types

5.1. DHCP on Bridge Ports 237

 VNS User Guide, Release 5.3.2

It is not mandatory to define a DHCP pool. The NSG will service DHCP requests differently depending on whether
such requests are received over a host or bridge vPort type:
• IP address assignment for DHCP request received on vPort Host:
– IP address assignment for host ports is handled by the VSD
– If a DHCP pool type “host” is defined then the host is assigned an IP address from the specified range.
– If not defined, then the host is assigned from the address range = subnet range - DHCP pool type “bridge”
• IP address assignment for DHCP request received on vPort Bridge:
– IP assignment is handled locally by NSG.
– If a DHCP pool type “bridge” is defined then the host is assigned an IP address from the specified range.
– If it is not defined, then any request on vPort Bridge will not be serviced.
There is a global DHCP IP lease expiry timer associated with the Organization profile that applies to all subnets defined
in the VSD. Default is 24 hours.

DHCP Decline Handling

When a DHCP address is allocated by the NSG to a client, the client may send a DHCP Decline message if it deter-
mines that the IP address is already in use by another host. The DHCP decline handling is as follows:
• On receipt of decline message, the offered IP address will be moved to a decline list for a non-configurable
duration of one hour provided no ARP is learnt with that IP from another port. After the hour timer expires, the
IP address is moved back to the available list.
• In case ARP is learned for a declined IP, that IP is moved to allocated list and when the ARP entry expires, the
IP will be moved to available list for DHCP allocation.

5.1.2 Provisioning Workflow

When a subnet is defined, either at template or instance level, a DHCP address pool for IP address alloca-
tion/management is specified. From the provisioned address range, either the entire subnet or a part of it can be
specified for IP address allocation via DHCP on a bridge or host ports.
In addition, at the subnet instance level, selected hosts (i.e. specified MAC address) can be assigned static IP addresses.
So if the DHCP request matches a configured MAC address, then the configured IP address is assigned to the host.

Define DHCP Pool

1. Login as CSPRoot or Enterprise Admin

2. In the Organization view, navigate to Networks -> (Layer 3 Domains ), and select a Domain -> Zone ->
Subnet for which DHCP Address Pool is to be defined.

3. Once a subnet is selected, click on the icon on the far right. The Address Ranges list is displayed (if already
defined). An existing address Range can be edited ( ) or a new address range can be added ( ) via the
popup New Address Range (page 239).

5.1. DHCP on Bridge Ports 238

 VNS User Guide, Release 5.3.2

Fig. 5.2: Specify Address Range for DHCP Pool

4. In the popup, specify Address Range by entering the First Address and the Last Address. In addition, the DHCP
Pool Type may be selected as Bridge or Host.

5. In order to create static binding of MAC addresses to IP addresses, select the icon for IP Reservations at
the top of the far right panel. Any existing static bindings will be listed and may be edited ( ) and/or new
bindings can be added ( ) using the New IP Reservation popup.

5.1. DHCP on Bridge Ports 239

 VNS User Guide, Release 5.3.2

Fig. 5.3: New IP Reservation Popup

6. In the popup, a valid MAC address and IP address from the subnet range can be entered to create the static
binding.

Modifying DHCP Address Ranges

Existing DHCP address ranges can be expanded or shrunk even when there are existing vPorts and host/bridge inter-
faces configured on the subnet. However, if static IP addresses would be impacted, address range expansion is not
permitted. If there are static IP bindings defined and an address range shrink is attempted, the VSD prompts the user
to confirm before proceeding. If the action is confirmed, the static IP bindings are automatically deleted.
If IP addresses fall outside of a shrunken address range, they are allocated a new IP address when the lease is renewed.
Consider the following scenarios regarding IP address reallocation. In the following scenarios, host1 is assigned IP9.
The address range 1-10 has been reduced to 2-5.
• When host1 renews the lease on IP9, the NSG sends a NAK message and offers a new available IP address from
the shrunken address range, such as IP2.
• When host1 renews the lease on IP9 and host2 has been assigned IP9, there is an IP conflict until the host1
renews its lease and is assigned a new IP.
• When host1 renews the lease on IP9, but a new address range 6-10 is defined, the lease is successfully renewed
and the IP address is not reallocated.
• Host1 renews the lease on IP9, but a new address range 6-10 is defined. A static reservation MAC2:IP9 is
defined, and host2 with the same MAC2 address requests an IP address. IP9 is assigned to host2, resulting in an
IP conflict. If host2 detects the duplicate IP and declines it, IP9 is no longer offered to host2. If host2 accepts
IP9, host1 is reallocated a new IP address.
Existing DHCP address ranges can be deleted only if there are no host or bridge interfaces configured on the subnet.
All interfaces on the subnet must be deleted before the address range can be deleted.
If vPorts are configured on the subnet with no interfaces, existing DHCP address ranges can be deleted or modified
without restriction.

5.1. DHCP on Bridge Ports 240

 VNS User Guide, Release 5.3.2

DHCP Lease Time Preference

DHCP lease times can be specified at the subnet or enterprise level. A DHCP lease time may also be requested by the
client. The preference order for the lease time to be used is as follows:
1. If specified, the lease time on the subnet is always preferred.
2. If a lease time is not specified on the subnet, the client lease time is preferred.
3. If neither the subnet nor the client specifies a lease time, the enterprise lease time is used.

Note: In the scenario where lease times are specified by the client and the enterprise, but not the subnet, the client-
requested lease time is used. However, if the client IP is deleted and the link flaps before requesting a new IP without
specifying a client lease time, the enterprise-specified lease time is not used. Instead, the last-known client-requested
lease time is still used.

DHCP Support

• DHCP leases are retained across OVS reboots

• DHCP leases are lost on vPort de-resolution

5.1.3 CLI Show Command Sample

*A:vsc# show vswitch-controller gateway dhcp 10.15.1.254

===============================================================================
DHCP Pool Range Table
===============================================================================
Start Address End Address Domain Name Subnet Name
-------------------------------------------------------------------------------
+----------------+------------------+-------------------+-----------------+
| 20.0.1.5 | 20.0.1.100 | ncpeDomain1 | ncpeEvpn2 |
+================+==================+===================+=================+
| 20.0.0.5 | 20.0.0.100 | ncpeDomain2 | ncpeEvpn4 |
+----------------+------------------+-------------------+-----------------+
| 20.0.6.5 | 20.0.6.100 | ncpeDomain2 | ncpeEvpn4 |
+----------------+------------------+-------------------+-----------------+
| 20.0.7.5 | 20.0.7.100 | ncpeDomain2 | ncpeEvpn4 |
+----------------+------------------+-------------------+-----------------+
-------------------------------------------------------------------------------
No. of DHCP Pool Ranges: 4
===============================================================================
===============================================================================
DHCP Static Mapping Table
===============================================================================
Ip Address Mac Address Domain Name Subnet Name
-------------------------------------------------------------------------------
20.0.6.99 ca:12:12:12:12:12 ncpeDomain2 ncpeEvpn1
-------------------------------------------------------------------------------
No. of Static DHCP Mappings: 1
===============================================================================

5.1. DHCP on Bridge Ports 241

 VNS User Guide, Release 5.3.2

5.2 Access Resiliency

• Overview (page 242)

• Terminology (page 242)
• Feature Description (page 243)
– RG Creation Considerations (page 244)
• Failure scenarios (page 244)
– Direct physical port failure on master NSG (page 244)
– Direct physical single LAN switch failure (page 245)
– Uplink failure/OF-TLS session loss on Master NSG (page 245)
– Failure in NSGs with dual uplinks (page 245)
• Provisioning Workflow (page 245)
– Create New Redundant Group (RG) (page 245)

* Modifying an existing RG (page 247)

* Deleting an existing RG (page 247)
– Add RG Ports (page 247)
– Add new VLANs to RG Ports (page 249)
– Add vPorts on RG Port (page 252)
• Interworking with other NSG features (page 253)
– Egress/Ingress QoS: (page 253)
– Port Address Translation: (page 253)
• CLI Show Command Sample (page 254)

5.2.1 Overview

The Nuage NSG connects the access or LAN side of an enterprise network and the WAN-facing network uplinks. The
current NSG hardware is deployed as a standalone unit with no in-built resiliency, and an enterprise may experience
widespread outage due to NSG or network link related failures. The access resiliency feature addresses resiliency of
the access side links of the NSG or alternatively can be viewed as port resiliency for the NSG. Access-side resiliency
feature enables NSGs co-located at a single customer site to be deployed as a redundant pair whereby one NSG is
designated authoritative and the other as secondary. In the event of a failure, the NSG pair serves as backup for each
other for traffic forwarding purposes.
Another feature called Dual Uplink provides redundancy for the network uplinks. The two features are complementary
and an organization may deploy any combination of single/redundant NSGs with single/dual uplinks.

5.2.2 Terminology

• Redundant Group(RG): A pair of NSGs form a RG.

• Master: Refers to the NSG entity that assumes the responsibility of forwarding the traffic at any given time.

5.2. Access Resiliency 242

 VNS User Guide, Release 5.3.2

• Backup: Refers to the NSG entity that will assume the responsibility of forwarding the traffic when the master
fails.
• Authoritative refers to the NSG or port that is configured to be the designated master.
• Secondary is standby or backup NSG or port.
• RG Member Designation: Configured as either “authoritative” or “secondary”
• Resilient Subnet: A subnet becomes resilient on attaching/resolving a resilient vPort to it. All hosts present in
the subnet are expected to be resilient and connect to the NSG via a switch or a router (i.e. not directly attached
to the NSG). Only the ports that are part of the RG can be associated with resilient subnets.
• RG Ports: Access ports that are members of an RG for offering protection during failure. Only the RG ports
can be associated with resilient subnets.

5.2.3 Feature Description

The feature enables two co-located NSGs’ deployed at a single customer site to be designated as a Redundant Group
(RG), whereby one NSG will act as a master NSG and the other acts as backup to provide access side resiliency. The
feature utilizes the detection of LAN side port-failure to trigger the mastership to change to the backup NSG. The
failure detection is implemented using a heartbeat mechanism between the NSGs. The capability is also supported for
the NSG-BR personality (page 178).

Fig. 5.4: Access Resiliency Topologies

The designation of a RG member as “authoritative” or “secondary” establishes:

• Configured role that will be assumed during election (master versus backup)
• Inheritance of VLANs and VPorts
• Ownership of configuration during RG decommissioning
If a user designated master (authoritative) NSG fails, the secondary NSG assumes traffic forwarding. When the
authoritative NSG recovers, there is an immediate switch over and it resumes traffic forwarding. However, it should be
noted that while switching from master and backup and vice versa, until BGP converges, some traffic loss is expected
on the network side.
The heartbeat mechanism is a per port keep-alive (BFD) between respective member ports to check for liveliness. The
BFD sessions are run over a control VLAN if the switch is VLAN-aware. If not, then the BFD sessions run using an
internally allocated IP address. BFD sessions run bi-directionally between master and backup NSGs in an RG.

5.2. Access Resiliency 243

 VNS User Guide, Release 5.3.2

Fig. 5.5: Control VLAN

The user configures:

• A VLAN number for the tagged control VLAN. It is recommended that the default of 4094 be maintained. If 0
is entered, then it will be treated as an untagged control VLAN.
• A heartbeat interval (default is 500 ms and maximum is 2000 ms). If there are 3 consecutive heartbeat failures,
then the switchover is triggered.
On port failure, the backup NSG becomes the master NSG for that port. All VLANS on the port fail over (entire
subnet fails over on port failure).
The NSGs periodically synchronize DHCP leases, so on a port failure the backup NSG will have the IP address
allocations/DHCP leases for the subnet failing over. Even though both NSGs have the DHCP state, the active DHCP
leases can only be seen on the master NSG (using ovs-appctl evpn/dhcp-table).

Note: Every 10s the DHCP lease information is exchanged between the RG NSGs.

RG Creation Considerations

• The RG NSGs can have different port counts. For instance, one NSG can have 6 ports and the other 8 ports.
• A resilient subnet can support a maximum of one vPort.
• NSGs can have more than one RG ports.
• Each RG NSG can have an active OF-TLS session established to the same or different VSC controller(s).
• A subnet cannot have a combination of resilient and non-resilient ports. In other words, a subnet may have either
one resilient port or multiple non-resilient ports.
• At any one time, only one RG NSG is forwarding for any subnet instance.
• RG NSGs cannot be inter-connected physically via a dedicated link.
• RG ports cannot be added to a RG with a single NSG.

5.2.4 Failure scenarios

Direct physical port failure on master NSG

• The backup NSG will detect a heartbeat protocol timeout and assume mastership by reporting the change to the
VSC. It is assumed that the vPorts on the backup NSG are already resolved.
• vPorts on the master NSG will be de-resolved.

5.2. Access Resiliency 244

 VNS User Guide, Release 5.3.2

• VSC will advertise the backup NSG as the next-hop for the LAN subnets
• Upon port recovery, once the heartbeat is established, the mastership is resumed by master NSG.
A direct physical port failure on the secondary NSG does not trigger any changes in VSC route advertisements. Also
the behavior is unchanged for NSG with single or dual uplink(s).

Direct physical single LAN switch failure

• Master and backup NSGs will de-resolve their respective vPorts

• VSC will withdraw the routes to LAN subnets.
• No LAN traffic will be forwarded
Behavior is unchanged for NSG with single or dual uplink(s).

Uplink failure/OF-TLS session loss on Master NSG

• Until vPorts are not de-resolved on the master NSG, and as long as the master NSG responds to the heartbeat
messages, master NSG retains mastership in the RG.
• Once the vPorts are de-resolved on the master NSG, backup NSG claims mastership.
• Once the OF-TLS session is reinitiated, the master assumes ownership of forwarding traffic.

Failure in NSGs with dual uplinks

• Uplink Failure/Loss of OF-TLS session on Master NSG

1. Single uplink failure will follow the behavior as supported by dual uplink feature.
2. Dual uplink failure: When vPorts are de-resolved on the master NSG, the heartbeats from the backup to
the master will fail, and the backup NSG will assume mastership by notifying the VSC.
3. On recovery of the uplinks, the old master will assume forwarding ownership.
• Uplink Failure/Loss of OF-TLS session on Backup NSG
1. No change on the master or VSC.
2. The vPorts will get de-resolved on the backup NSG and the heartbeat from master to backup will fail.

5.2.5 Provisioning Workflow

Create New Redundant Group (RG)

An RG can be created out of two instantiated gateways by a CSPRoot or enterprise administrator at the organization
level. In this step, the user designates an authoritative and a secondary NSG.

5.2. Access Resiliency 245

 VNS User Guide, Release 5.3.2

Fig. 5.6: Create Redundant Group

1. Select two NSGs and click on the icon which will open the New Redundancy Group popup.
2. Enter a Name and Description for the RG.
3. The selected NSGs will get designated as Authoritative and Secondary. The designated roles can be switched
using by clicking the little X icon in the middle of the popup, between the peers. If the group is created with one
NSG selected, then it will be designated as Authoritative, and the user will have to associate a secondary NSG
separately.
4. Enter a VLAN number for the Heartbeat VLAN (default is 4094). This will serve as a reserved VLAN used
for hearbeats for each resilient physical access port. Enter 0 to create an untagged VLAN.

Note: If you attempt to change a tagged control VLAN to untagged, then any existing VLANs on RG ports will have
to be deleted before you can proceed.

5. Enter a Heartbeat Interval in milliseconds to declare the master dead. Defaults is 500 ms. Valid range is 500

5.2. Access Resiliency 246

 VNS User Guide, Release 5.3.2

ms to 2000 ms. After three consecutive failures, the backup NSG declares itself as master.
6. Click Create to form the RG.

Note:
• The RG group can have a bootstrapped and a non-bootstrapped NSG to form a RG.
• The non-bootstrapped NSG will need to be bootstrapped before it can participate in any services, which means
that the non-bootstrapped NSG can only act as a secondary.
• An RG with one NSG is not permitted.
• On breaking up an RG group, the master NSG retains the service configuration while the service configurations
are removed from the secondary NSG.
• RG ports have to be added to the RG pair to create resilient ports.

Modifying an existing RG

1. To modify a redundancy group, either right click the group and select Edit, or select the group and click the
pencil icon at the bottom of the panel.
2. To change both NSGs in an RG group, the existing group has to be deleted and a new RG group with the selected
NSGs must be created.
3. Only a secondary NSG may be removed or replaced from an RG. However, the secondary NSG should not have
vPorts on the RG ports resolved. Upon removal, the service configuration on the secondary NSG will be deleted.
4. The user can switch the designated roles of the authoritative/secondary NSG. The designated roles can be
switched using by clicking the little icon in the middle of the popup, between the peers.

Note: A role switchover is service impacting and user will be warned.

Deleting an existing RG

To delete a redundancy group, either right click the group and select Delete, or select the group and click the minus
icon (-). To complete the operation, click the red confirm button.

Note: When a RG is deleted:

• Master NSG will retain the configuration and continue to service the vPorts that were on the RG ports.
• Backup NSG will retain the configuration but vPorts on the RG ports will be removed.
• vPorts on non-RG ports will be intact.
• Control VLANs will also be deleted from the RG ports on both NSGs.

Add RG Ports

1. Select an RG and select click the plus sign (+) at the bottom of second panel from the left for Redundant Port.

5.2. Access Resiliency 247

 VNS User Guide, Release 5.3.2

Fig. 5.7: Add RG Port

5.2. Access Resiliency 248
 VNS User Guide, Release 5.3.2

2. The New Redundant Port popup appears. The popup will list all the available common ports. Select ports from
the list that will be part of the RG and click Create. After the creation of the first port, it will be displayed in the
panel, and you can right click it to add additional ports, edit, or delete them or continue to use the plus sign.

Fig. 5.8: Select RG Ports

Note: An RG port will appear in the individual gateway instance as a “read only” port.

If the port profiles of the selected RG ports are different, then the port profiles for the secondary NSG will be over-
written with the port profiles of the authoritative NSG. A warning will be displayed that the port profiles for protected
ports are different and will overwritten during the next configuration window.

Note: While adding a port to RG, it should not have any vPorts on it.

Add new VLANs to RG Ports

1. Select a RG Port and select click the plus sign (+) at the bottom of third panel from the left for VLANs. When
a RG port is created, a control VLAN is displayed as read-only and cannot be modified by a user.

5.2. Access Resiliency 249

 VNS User Guide, Release 5.3.2

Fig. 5.9: Add VLANs Page

2. A new VLAN popup appears. Only VLAN numbers that are not the same as the control VLAN will be accepted.

5.2. Access Resiliency 250

 VNS User Guide, Release 5.3.2

Fig. 5.10: New VLAN Popup

3. On the far right panel is a checkbox to create an untagged control VLAN for the selected port. When the
checkbox is selected, the tagged control VLAN will be automatically deleted. However, if there are any existing
VLANs on the port, then an error will be displayed when Update (at the bottom of the screen) is clicked to
complete the operation. Thus, it is recommended that you first remove any existing VLANs and then initiate the
creation of the untagged control VLAN.

5.2. Access Resiliency 251

 VNS User Guide, Release 5.3.2

Fig. 5.11: Create Untagged Control VLAN

Note: The switch ports option can be used to switch the master/backup roles of the NSGs for the selected port.

Add vPorts on RG Port

1. This is similar to creating VPorts on an NSG. However instead of selecting an NSG, a RG must be selected in
the workflow.

5.2. Access Resiliency 252

 VNS User Guide, Release 5.3.2

Fig. 5.12: Add vPorts on RG Port

Note:
• RG port member vPorts belonging to a secondary NSG do not forward/respond to arp etc.. All traffic will be
dropped in secondary NSG.
• vPorts are created on both NSGs.
• Host and Bridge vPorts of a subnet cannot be present in both resilient and non-resilient ports.

5.2.6 Interworking with other NSG features

Important: Refer to the Nuage VNS Release Notes for all the supported use case scenarios involving interactions
with features such as dual-uplink etc..

Egress/Ingress QoS:

• The ports that are part of the RG do not automatically inherit the ingress QOS profiles from the authoritative
NSG.
• The QOS profiles need to be attached to the P/V constructs for both authoritative and secondary NSGs.

Port Address Translation:

For resilient subnets that have PAT enabled, there are two cases:

5.2. Access Resiliency 253

 VNS User Guide, Release 5.3.2

• PAT to the uplink IP: The source IP of the outgoing packet will be the uplink WAN IP address of the NSG that
is operationally the master. If the NSG has two uplinks, then the dual uplink feature will dictate which uplink
gets selected.
• PAT Pools: The source IP of the outgoing packet will be the configured IP address from the pool specified for
the NSG that is operationally the master.

Note: It is expected to have two independent PAT pools per NSG forming the RG pair. PAT provisioning workflow
will remain unchanged as a result of this feature.

5.2.7 CLI Show Command Sample

*A:vsc# show vswitch-controller gateway ports

===============================================================================
Gateway Ports Table
===============================================================================
Gateway IP Port Mode Vlan VSD Gateway Heartbeat Heartbeat
Name range Role Role Vlan Interval
-------------------------------------------------------------------------------
33.97.128.188 port2 access 0-4094 n/a n/a n/a n/a
33.97.128.188 port3 access 0-4094 n/a n/a n/a n/a
33.97.128.188 port4 access 0-4094 n/a n/a n/a n/a
33.97.128.188 port5 access 0-4094 n/a n/a n/a n/a
33.97.128.188 port6 access 0-4094 n/a n/a n/a n/a
125.129.113.35 port2 access 0-4094 n/a n/a n/a n/a
125.129.113.35 port3 access 0-4094 master master 4094 500
125.129.113.35 port4 access 0-4094 master master n/a 500
125.129.113.35 port5 access 0-4094 n/a n/a n/a n/a
125.129.113.35 port6 access 0-4094 n/a n/a n/a n/a
126.152.224.116 port2 access 0-4094 n/a n/a n/a n/a
126.152.224.116 port3 access 0-4094 backup backup 4094 500
126.152.224.116 port4 access 0-4094 backup backup n/a 500
126.152.224.116 port5 access 0-4094 n/a n/a n/a n/a
126.152.224.116 port6 access 0-4094 n/a n/a n/a n/a
-------------------------------------------------------------------------------
No. of Ports: 15
===============================================================================

5.2. Access Resiliency 254

 VNS User Guide, Release 5.3.2

5.3 Port Address Translation

• Overview (page 255)

• Use Case Scenarios (page 257)
• NSG Forwarding Behavior (page 258)
• Per Uplink NAT (page 259)
• Overlapping PAT Pools (page 259)
• Address Pool Considerations (page 259)
• Provisioning Workflow (page 260)
– Enabling PAT (page 260)
– Configuring Address Translation Pools (page 262)

* Creating Address Translation Pools (page 263)

* Assigning Address Translation Pools to NSG or Network Port VLAN (page 264)
* Configuring Address Maps (page 265)
– Advertising Address Pools (page 266)
• PAT Statistics (page 267)
– Displaying PAT Statistics on VSD UI (page 267)

* Address Translation Pool Level Statistics (page 267)

* Static Address Map Level Statistics (page 270)
* Port/VLAN Level Statistics (page 271)
• CLI Show Command Sample (page 272)

5.3.1 Overview

The Port Address Translation (PAT) feature addresses local Internet offload uses cases when the Nuage NSG is de-
ployed as a WAN router for connectivity to business Internet services. The following scenarios are supported whereby
LAN subnet traffic is routed to the underlay for Internet access instead of connecting to an overlay VPN:
• Private LAN subnets are not Internet routable and therefore must be mapped to a single routable public IP
addresses which get advertised to upstream PEs.
• Private LAN subnets host certain services that must be accessed from external networks, and therefore must be
mapped 1:1 to routable public IP addresses and allow incoming connection requests even when the internal host
has not initiated a request.
• LAN-side hosts have routable public IP addresses, therefore traffic from such hosts is required to be sent to the
underlay without PAT.
In order to address all the local Internet offload use cases, the following address translation configurations are offered:
• PAT where private user/application IP addresses are mapped to a single external routable address (referred to as
PAT IP). The single external routable address can be the IP address of the WAN uplink or an IP address that is
specified from a configured address pool. If the PAT IP is not defined, it defaults to the WAN uplink IP address
(provided the behavior is enabled via configuration).

5.3. Port Address Translation 255

 VNS User Guide, Release 5.3.2

• 1:1 NAT (or Static Source NAT) where private IP address of an external facing user/application is statically
mapped to an external routable address.
• N:1 PAT (or Static Port Forwarding) where private IP address and port of an external facing user/application is
statically mapped to an external routable address and port. This is more restrictive than static 1:1 NAT capability
because only certain ports are exposed to public as compared to all ports.
• Dynamic Source NAT (DSNAT) where private IP address of any external facing user/application in a subnet is
dynamically mapped to an external routable address in the defined address pool. In other words, with DSNAT,
hosts in an NSG subnet will dynamically be assigned outside IP addresses from the pool, and not be limited to
a single IP (with a maximum of 65000 ports).
DSNAT feature must be explicitly enabled, via a flag, when the address pool is defined. The pool for which the flag
is enabled is referred to as a DSNAT pool. The entire DSNAT pool range is added by the VSC into the BGP underlay
as individual /32 routes pointing back to the NSG. The default conntrack timeouts are used to clear out the dynamic
address translations to free up the IP addresses for re-assignment.
Address pools are created by the CSPRoot and assigned to enterprises for consumption.

Fig. 5.13: NSG Forwarding Pipeline and Address Pools

Note: Release 4.0.R1 onward, the address pools have a per network port/VLAN scope to allow for enterprise owned
portable pools. Any existing configurations with pools assigned to NSG instances will continue to be supported from
a forwarding perspective.
However, to take advantage of the port translation feature enhancements introduced in 4.0.R1, it is recommended that
the any pools assigned to NSG instances be delinked and equivalent pools using a begin-end range format be associated
with that same NSG instance or with the network port/VLAN of the NSG instance instead.

Address translation applies to various gateway deployment scenarios:

• Single NSG, single up-link
• Single NSG, dual up-link
• Redundant NSG Pair

Note: Address translation is not supported with PPPoE on uplink.

Note: PAT pools with default source IPs in the same subnet as a PPPoE uplink IP are not supported.

5.3. Port Address Translation 256

 VNS User Guide, Release 5.3.2

5.3.2 Use Case Scenarios

The following Local Internet Offload use scenarios are supported by configuring the necessary parameters.

Note: Following example scenarios assume single uplink with address pool assigned to untagged VLAN of Port 1
(referred to as Port 1 for readability in each scenario). For dual uplinks, the address pools will be per port/VLAN and
default forwarding behavior of Primary over Secondary will apply.

Scenario 1: Customer wants all domain service traffic to go directly to the Internet using the WAN uplink on Port 1.
Configure as follows:
Domain Address Translation Support Enabled
Subnet Address Translation Support Enabled or Inherited
Domain Underlay Support Enabled
Subnet Underlay Support Enabled or Inherited
Port 1 Address Pool Not Configured
Scenario 2: Customer wants all domain service traffic to go directly to the Internet using a public IP address that is
not the WAN uplink IP address. This would be the case if the customer wants to keep the transport provider’s address
separate from the public address for service routing. It is assumed that there is a route, provisioned by the WAN
transport provider, to reach the non-WAN public IP address. Configure as follows:
Domain Address Translation Enabled
Support
Subnet Address Translation Enabled or Inherited
Support
Domain Underlay Support Enabled
Subnet Underlay Support Enabled or Inherited
Port 1 Address Pool Configured with a default PAT pool IP address (non-WAN uplink IP
address)
Scenario 3: Customer wants to locally offload traffic from some subnet(s) in a domain while preserving the source IP
address. Configure as follows:
Domain Address Translation Support Enabled
Subnet Address Translation Support Disabled
Domain Underlay Support Enabled
Subnet Underlay Support Enabled or Inherited
Port 1 Address Pool Not Applicable
Scenario 4: Customer wants to invoke PAT’ing for specific subnet(s) in a domain allowing traffic to go directly to the
Internet using the WAN uplink IP address. Configure as follows:
Domain Address Translation Support Disabled
Subnet Address Translation Support Enabled
UDomain Underlay Support Enabled
Subnet Underlay Support Enabled or Inherited
Port 1 Address Pool Not Configured
Scenario 5: Customer wants to create 1:1 NAT mapping using public IP addresses that are not the WAN uplink address
or the default pool IP address. It is assumed that there is a route, provisioned by the WAN transport provider, to reach
all such public IP addresses. Configure as follows:

5.3. Port Address Translation 257

 VNS User Guide, Release 5.3.2

Domain Address Translation Disabled

Support
Subnet Address Translation Enabled
Support
Domain Underlay Support Enabled
Subnet Underlay Support Enabled or Inherited
Port 1 Address Pool Configured with or without default IP. Additional NAT maps must be
defined
Scenario 6: Customer wants to define static port forwarding to allow WAN to LAN connection requests using public
IP addresses and ports. It is assumed that there is a route, provisioned by the WAN transport provider, to reach all such
public IP addresses. Configure as follows:
Domain Address Translation Support Disabled
Subnet Address Translation Support Enabled
Domain Underlay Support Enabled
Subnet Underlay Support Enabled or Inherited
Port 1 Address Pool Configured with or without default IP. N:1 PAT maps must be defined
Scenario 7: Customer has public IP hosts behind the NSG and traffic from the hosts must be directly offloaded to the
Internet without any PAT’ing. It is assumed that there is a route, provisioned by the WAN transport provider, to reach
all such hosts. Configure as follows:
Domain Disabled
Subnet Address Translation Support Disabled
Domain Underlay Support Enabled
Subnet Underlay Support Enabled or Inherited
Port 1 Address Pool Not applicable

5.3.3 NSG Forwarding Behavior

The feature functionality is triggered by several configuration settings, specifically whether:

• Address pool support is enabled/disabled at domain and subnet level
• Underlay support is enabled/disabled at domain and subnet level, where:
– Disabled ensures that if a route is not found in the overlay, then the traffic is dropped.
– Enabled ensures that if a route is not found in the overlay, then the traffic is forwarded to the underlay.

Note: When traffic from any subnet is to be tunnelled via the overlay, the Subnet Underlay Support flag MUST be
set to Disabled. The reason is that if a route it not found in the overlay, then the desired behaviour for such traffic is
that it be dropped, and not be inadvertently offloaded to the Internet.

Assuming that address pool support and underlay support is enabled at domain and inherited at subnet level, forwarding
is as follows (assuming single uplink and address pool assigned to NSG):
1. For traffic sourced from the LAN destined to WAN
• If address translation pool is not configured, then the NSG will PAT to the underlay using the network uplink
port IP address
• If address translation pool is configured, then the forwarding is determined by matching port mapping rules in
the following order:
– 1:1 NAT

5.3. Port Address Translation 258

 VNS User Guide, Release 5.3.2

– Dynamic Source NAT

– PAT to underlay using PAT IP (which is the default IP if defined or uplink port IP)
2. For connection requests from WAN destined to LAN:
• If address translation pool is configured, then the forwarding is determined by matching port mapping rules in
the following order:
• N:1 PAT
• 1:1 NAT
• else drop

5.3.4 Per Uplink NAT

Underlay routing and NAT can be enabled at the uplink level. In a dual uplink scenario, this means that one uplink
can be configured for underlay routing to the MPLS WAN and one uplink can be configured for underlay NAT to the
Internet.
Per uplink NAT is supported as part of the SaaS routing solution See Per uplink NAT (page 566) in the “SaaS routing”
chapter for more information.

5.3.5 Overlapping PAT Pools

Address pools with exactly or partially overlapping private IP address ranges are supported and can be assigned to
uplinks on different NSGs. This allows you to use 1:1 or N:1 address maps when the address pool contains private IP
addresses within a subnet that is reused across multiple NSGs.
Consider the following when configuring PAT pools with overlapping IP ranges:
• Overlapping address pools can be assigned to two uplinks on the same NSG.
• Overlapping address pools are supported with redundancy groups only with BGP configured on the uplinks and
with both NSGs on different CPEs.
• Although identical address pools can be created and assigned to NSGs in VSD, the same address pool object
cannot be assigned to multiple NSGs.

Note: Overlapping PAT pools are intended to be used with private IP address ranges or on disjoint underlays.
However, the VSD does not prevent you from configuring overlapping public IP address ranges. These public ranges
would be advertised by BGP on the WAN or from the VSC.

5.3.6 Address Pool Considerations

Consider the following when defining address ranges for NAT or PAT pools:
• There can be multiple pools associated with a single uplink port/VLAN instance, provided the address ranges
do not overlap.
• Separate pools can be assigned to each uplink on a dual uplink NSG.
• A maximum of one PAT IP can be defined across all pools assigned to one uplink port/VLAN instance. Dual
uplink NSGs can have up to two PAT IPs - one per uplink.

5.3. Port Address Translation 259

 VNS User Guide, Release 5.3.2

• A host (located in a LAN) can have only one 1:1 NAT mapping entry across all the pools assigned to a single
uplink.
• A LAN-side host can have up to two 1:1 NAT entries - one per pool per port.
• A host can have multiple N:1 PAT entries in one or more pools regardless of which port instance the pools are
attached to.

5.3.7 Provisioning Workflow

Enabling PAT

The feature can be enabled or disabled at the domain level. By default it is disabled at domain level.

Note: It is strongly recommended that PAT and Underlay Support be enabled at the domain level to allow the
enablement of PAT at subnet level once vPorts are resolved.

If PAT is enabled at the domain level, then all subnets are eligible to be PATed by inheritance (default). PAT may be
enabled or disabled selectively at subnet level.
The following workflow describes how PAT and Underlay Support can be enabled/disabled at the Domain and Subnet
Instance level.

1. In the Organization view, navigate to Networks -> Layer 3 Domains ( ) -> Design. Select an instantiated
domain displayed in the the Layer 3 Domains panel.
2. Once the domain is selected, the feature can be enabled or disabled using the Domain level parameter “Address
Translation Support” on the right side of the screen in the Address Translation sub-section as shown in the figure
Domain Level Parameters (page 261). The parameter can be set to Enabled or Disabled. Default is disabled.

5.3. Port Address Translation 260

 VNS User Guide, Release 5.3.2

Fig. 5.14: Domain Level Parameters

3. For the selected domain, the “Underlay Support” parameter can be set to enabled (local offload) or disabled
(drop). The default is disabled.
4. On the selected domain, the topology can be expanded to display the defined zone and subnets. By selecting
a specific subnet, the subnet level parameter for “Address Translation Support” can be set to either Inherited,
Enabled, or Disabled as shown circled in Subnet Level Parameters (page 262) . The default is inherited which
means that it will inherit the Domain level setting. Otherwise the user can select an explicit Enabled or Disabled.

5.3. Port Address Translation 261

 VNS User Guide, Release 5.3.2

Fig. 5.15: Subnet Level Parameters

5. For the selected subnet, the “Underlay Support” parameter can be set to enabled or disabled. The default is
inherited which means that it will inherit the Domain level setting.

Configuring Address Translation Pools

• At CSP level, CSPRoot creates Address Translation Pools

• CSPRoot assigns one or more pool to enterprise for consumption
• Within an enterprise, at the gateway instance level the pool is associated with the VLAN of the network port.

Note: Prior to 4.0.R1, the pools had a gateway instance scope. It is recommended that the address pools be unlinked
from the NSG instances and added to network port(s) VLAN.

• A network port VLAN can have multiple pools assigned. However, at most one pool must have a default IP
configured.
• 1:1 NAT and N:1 PAT entries can be added once an address pool is assigned to an instance. The workflow is
described in more detail as follows:

5.3. Port Address Translation 262

 VNS User Guide, Release 5.3.2

Creating Address Translation Pools

1. User is logged in as CSProot.

2. Click on the Data Center Configuration icon .

3. Select the Infrastructure tab, and click on Address Translation Pools icon . The window displays a list of
existing address pools (if they exist). Click on towards the bottom of the screen to create a new address pool
which appears as a popup displayed in Figure Defining a new Address Translation Pool (page 263).

Fig. 5.16: Defining a new Address Translation Pool

4. Enter the Name, Description, a pool of IP addresses, and optionally a Port Address Translation IP (default
IP) address from the pool that will be used for PAT.

5.3. Port Address Translation 263

 VNS User Guide, Release 5.3.2

It is recommended that the default IP be defined by the Enterprise admin when assigning the pool to the network port
VLAN. In case a default IP address is specified by CSPRoot, then the enterprise administrator may override it.
The address pool is specified as a range using a start address (First IP) (e.g. 135.10.0.24) and an end address (Last
IP) (e.g. 135.10.0.34). The entire range, inclusive of First:Last IP addresses will be used.

Note:
• The First:Last IP addresses must be unicast IP addresses and cannot be part of special purpose IP ranges as
specified in RFC6890 (0/8, 127/8, 169.254/16, 224/4, 240/4, 255.255.255.255/32).
• A maximum of 16 IPs can be defined in any pool, with a maximum of 32 across all pools.

Note: If the address pool is in a different subnet than the WAN uplink IP address, then refer to Advertising Address
Pools (page 266) for ensuring that return traffic can be correctly routed from the network to the configured address
range.

5. For Enterprise Address Translation Pool Assignment, click on the icon which pops up a list of existing
enterprises, from which one enterprise can be selected. CSPRoot can assign more than one address pool to an
enterprise.
6. Check Dynamic Source NAT (if DSNAT capability is desired) and specify the source address range using the
Start Source Address and End Source Address fields. A pool with DSNAT flag enabled is referred to as
DSNAT pool.
If the start and end source addresses are not specified, then there is no specific source matching and will serve as
default source match.

Note:
• There can be at most one DSNAT pool with both start and end source addresses not specified.
• If DSNAT is enabled, then default PAT IP is not applicable and will not be allowed.
• If DSNAT is enabled, then address map creation is also not applicable and not allowed.

Assigning Address Translation Pools to NSG or Network Port VLAN

1. At the Enterprise level, the CSPRoot or the Enterprise administrator can assign one or more of the created
address pools to an NSG or network port VLAN.

2. For assigning pool to NSG, select the NSG instance by navigating to Infrastructure -> NSG . On the far
right, click on the icon for associating address pools. If a pool is already assigned, it will be listed in the
column below the icon.

3. For assigning pool to network port VLAN, first select the NSG by navigating to Infrastructure -> NSG .
From the Ports panel select the network port (port1 or port2) and the VLAN from the VLAN panel. On the far
right, click on the icon for associating address pools. If a pool is already assigned, it will be listed in the
column below the icon.

5.3. Port Address Translation 264

 VNS User Guide, Release 5.3.2

4. Depending on whether it is a first time assign or not, either icon or icon will be displayed towards the
lower part of the panel. Clicking on the icon will pop up a list of address pools available (for the enterprise) for
consumption as shown in Assigning an address pool to network port VLAN (page 265)

Note:
• Multiple pools may be assigned, but a maximum of one pool may have a default IP address configured. So if
you assign one pool with a default IP address, and then attempt to assign another pool that also has a default IP
address, you will receive a validation error and will not be able to proceed.
• When assigning pools, you can have only one pool with default source match, that is either a regular pool with
default IP or a DSNAT pool with both start and end addresses not specified.

Fig. 5.17: Assigning an address pool to network port VLAN

Configuring Address Maps

1. The CSPRoot or the enterprise admin can also define 1:1 NAT map and N:1 PAT map from an address pool.
2. Navigate to the address pools specified for the enterprise, by selecting the Infrastructure tab, and clicking on
Address Translation Pools icon . The window displays a list of existing address pools (if they exist).

3. Select an address pool and click on icon or icon in the Address Maps panel of the screen which will
popup a New Address Map screen in which the static binding (1:1 or N:1) can be configured using IP addresses

5.3. Port Address Translation 265

 VNS User Guide, Release 5.3.2

from the pool.

Fig. 5.18: 1:1 NAT Map

Fig. 5.19: N:1 PAT Map

Advertising Address Pools

There are two options for ensuring that return traffic can be correctly routed from the network to the configured address
range:
1. Static routes
Configure a static route on the upstream router for the address pool subnet with the NSG uplink IP address as next-hop.
2. Route Advertisements from VSC

5.3. Port Address Translation 266

 VNS User Guide, Release 5.3.2

The VSC leaks the default IP, and the public IP’s from 1:1 NAT maps and N:1 PAT maps (referred to as “PAT Pool
routes” hereafter) into the base routing table, with the NSG uplink IP as the next-hop. These routes are installed in the
VSC route table as “Remote Static” routes with the lowest preference of 255.
To advertise the PAT Pool routes to BGP peered PE routers, the Remote Static routes can be exported as IPv4 routes
using BGP import policies (refer to 7750 user guide).
Pools associated with a network port VLAN:
The PAT pool routes will be leaked with a next-hop that equals the uplink IP address of the network port VLAN the
pool is associated with.
The VSC will leak the PAT pool routes for the address pool(s) associated with the network port from which the open
flow connection is established.
Pools associated with an NSG:
In the case of dual uplinks, the address pool is reachable from both uplinks, all connected VSCs leak the PAT Pool
routes in their base routing table with both uplink IPs as next-hops.

5.3.8 PAT Statistics

PAT statistics collection is enabled by default when PAT is enabled at the domain level.
The PAT statistics may be:
• Viewed graphically via the Statistics display window on the VSD UI,
• Retrieved using CLI on the primary VSC
• Accessed via VSD ReST API.

Note: For network ports, the collection interval is set to 60 seconds and is not configurable.

PAT Statistics are available at:

1. Address Translation Pool level for:
1. Pools with Dynamic Source NAT disabled (provides a view for all configured static address maps)
2. Pools with Dynamic Source NAT enabled
2. Static Address Map level
3. Untagged VLAN level for traffic PAT’ed via default IP and Uplink IP.
Packet and Byte count views are available for Egress (Out) and Ingress (In) traffic (as applicable). Egress refers to
traffic from gateway to network and Ingress refers to traffic from network to the gateway.

Displaying PAT Statistics on VSD UI

Address Translation Pool Level Statistics

An address translation pool which can be selected using either of the following two navigation options:

• Navigate to Infrastructure -> NSG -> Address Translation Pool ), Select a pool and click on the statistics
button associated with the pool.

5.3. Port Address Translation 267

 VNS User Guide, Release 5.3.2

Fig. 5.20: Select Address Translation Pool for Statistics

• Navigate to Infrastructure -> NSG instance -> NS Ports -> VLANs, and select address pool icon on the
upper right, which will display a list of available pools in the right most panel. Select a pool and click on the
statistics button associated with the pool.

Fig. 5.21: Select Address Translation Pool for Statistics

1. If the selected pool has Dynamic Source NAT disabled, then the statistics window displays all the configured
address maps and provides a Bytes or Packets view of ingress (In) and egress (Out) traffic for one or more
address map(s).

5.3. Port Address Translation 268

 VNS User Guide, Release 5.3.2

Note: If Ingress/Egress rate is same, then the color overlaps and only one colors is displayed. Select In or Out to view
individual statistics for ingress/egress.

Fig. 5.22: Statistics at Address Pool (for selected static map)

2. If the selected pool has Dynamic Source NAT enabled, then the statistics window displays a combined count
for all DSNATed traffic (all egress traffic which is source NAT’ed via the Dynamic Source NAT IP tables rule).
A Bytes or Packets view may be selected for egress (Out) statistics.

5.3. Port Address Translation 269

 VNS User Guide, Release 5.3.2

Fig. 5.23: Statistics for Dynamic Source NAT-enabled Pool (Packets View)

Static Address Map Level Statistics

Navigate to Infrastructure -> NSG -> Address Translation Pool ) -> Address Maps, and select a specific map.
Click on the statistics button associated with the selected static address map. A Bytes or Packets view may be
selected for ingress (In) or egress (Out) statistics.

Note: If Ingress/Egress rate is same, then the color overlaps and only one colors is displayed. Select In or Out to view
individual statistics for ingress/egress.

5.3. Port Address Translation 270

 VNS User Guide, Release 5.3.2

Fig. 5.24: Statistics for Specific Static Address Map (Bytes View)

Port/VLAN Level Statistics

Navigate to Infrastructure -> NSG instance -> NS Port -> VLANs, and select the untagged VLAN for the network
port. Click on the statistics icon associated with the VLAN0. A Bytes or Packets view may be selected for egress
(Out) statistics for traffic PAT’ed to Default IP or Uplink IP (as applicable).

5.3. Port Address Translation 271

 VNS User Guide, Release 5.3.2

Fig. 5.25: Statistics for Port/VLAN (Packets View)

5.3.9 CLI Show Command Sample

1. NSG Datapath Id is 154.78.127.221 and Dut-G is the VSC on which the command is run:

*A:Dut-G# show vswitch-controller gateway ports 154.78.127.221 pat

===============================================================================
Gateway Ports PAT Information Table
===============================================================================

Port Mode : network Port Name : port1

PAT Config :
Default PAT IP : 7.7.7.1
-------------------------------------------------------------------------------
PAT Pool UUID : 07d526cc-20a7-4d9b-836d-76be20f6c1d1
PAT Pool : 7.7.7.1 to 7.7.7.16
NAT Map Info :
NAT Map UUID : 431dba30-1dd2-4d26-b142-c4953d02a370
Private IP address : 20.0.0.3 Public IP address : 7.7.7.3
Private Port : 3000 Public Port : 33000
NAT Map Type : N:1 PAT

NAT Map UUID : c5a320aa-b40f-47c4-b410-2b279ee84219

Private IP address : 20.0.0.2 Public IP address : 7.7.7.2
NAT Map Type : 1:1 NAT
-------------------------------------------------------------------------------

Port Mode : network Port Name : port2

5.3. Port Address Translation 272

 VNS User Guide, Release 5.3.2

PAT Config :
Default PAT IP : 8.8.8.10
-------------------------------------------------------------------------------
PAT Pool UUID : 13347338-4133-4ed7-81a3-6775bf4c190b
PAT Pool : 8.8.8.1 to 8.8.8.10
NAT Map Info :
NAT Map UUID : 5ee4fdcd-fcf5-4369-a372-5784cd108a8b
Private IP address : 20.0.0.5 Public IP address : 8.8.8.5
Private Port : 2000 Public Port : 20000
NAT Map Type : N:1 PAT

NAT Map UUID : 6a5a8f0b-4569-40e7-b48b-66a3b717da33

Private IP address : 20.0.0.4 Public IP address : 8.8.8.4
NAT Map Type : 1:1 NAT
-------------------------------------------------------------------------------
No. of Ports: 2
===============================================================================

2. For a particular uplink, port pool information:

*A:Dut-G# show vswitch-controller gateway ports 154.78.127.221 pat port-name "port2"

===============================================================================
Gateway Ports PAT Information Table
===============================================================================

Port Mode : network Port Name : port2

PAT Config :
Default PAT IP : 8.8.8.10
-------------------------------------------------------------------------------
PAT Pool UUID : 13347338-4133-4ed7-81a3-6775bf4c190b
PAT Pool : 8.8.8.1 to 8.8.8.10
NAT Map Info :
NAT Map UUID : 5ee4fdcd-fcf5-4369-a372-5784cd108a8b
Private IP address : 20.0.0.5 Public IP address : 8.8.8.5
Private Port : 2000 Public Port : 20000
NAT Map Type : N:1 PAT

NAT Map UUID : 6a5a8f0b-4569-40e7-b48b-66a3b717da33

Private IP address : 20.0.0.4 Public IP address : 8.8.8.4
NAT Map Type : 1:1 NAT
-------------------------------------------------------------------------------
No. of Ports: 1
===============================================================================

3. Display PAT pool routes in the underlay. Assumption is that the VSC and Uplink Router are BGP peers and
capable of exchanging “IPv4” BGP routes.
In these examples, Dut-G is the VSC, Dut-F is the uplink router and Dut-I (10.20.1.9) is the BGP RR (route-reflector);
that is both the VSC and uplink router are BGP peers with the RR.
Step 1: The VSC (controller Dut-G) leaks these routes in the underlay route table with the respective uplink next-hop.

*A:Dut-G# show router route-table

===============================================================================
Route Table (Router: Base)
===============================================================================

5.3. Port Address Translation 273

 VNS User Guide, Release 5.3.2

Dest Prefix[Flags] Type Proto Age Pref

Next Hop[Interface Name] Metric
-------------------------------------------------------------------------------
7.7.7.1/32 Remote Static 00h15m29s 255
10.15.1.254 0
7.7.7.2/32 Remote Static 00h15m29s 255
10.15.1.254 0
7.7.7.3/32 Remote Static 00h15m29s 255
10.15.1.254 0
8.8.8.4/32 Remote Static 00h09m22s 255
10.18.1.254 0
8.8.8.5/32 Remote Static 00h09m22s 255
10.18.1.254 0
8.8.8.10/32 Remote Static 00h09m22s 255
10.18.1.254 0
--------------------------------------------------------------------------------------
˓→------------------------------------------

Step 2: User must configure a BGP policy on the VSC to export static-routes via BGP IPv4 and then apply the policy
under BGP:

*A:Dut-G# configure router policy-options

*A:Dut-G>config>router>policy-options# info
----------------------------------------------
policy-statement "VSPnoTunnelsBGPPolicy"
entry 10
from
protocol static
exit
to
protocol bgp
exit
action accept
exit
exit
exit
----------------------------------------------

*A:Dut-G# show router policy "VSPnoTunnelsBGPPolicy"

entry 10
from
protocol static
exit
to
protocol bgp
exit
action accept
exit
exit

*A:Dut-G>config>router# bgp
*A:Dut-G>config>router>bgp# info
----------------------------------------------
family ipv4 vpn-ipv4 evpn
connect-retry 5
hold-time 120
min-route-advertisement 1
export "VSPnoTunnelsBGPPolicy"

5.3. Port Address Translation 274

 VNS User Guide, Release 5.3.2

router-id 10.20.1.7
rapid-withdrawal
rapid-update evpn
group "test"
neighbor 10.20.1.9
med-out 100
peer-as 1000
exit
exit
no shutdown
----------------------------------------------

Step 3 : The pat-pool routes (/32s for the default IP and public IPs in the address maps) should now be available on the
uplink router (Dut-F in this example) pointing to the correct respective next-hop routing towards the respective uplink:

*A:Dut-F# show router route-table

===============================================================================
Route Table (Router: Base)
===============================================================================
Dest Prefix[Flags] Type Proto Age Pref
Next Hop[Interface Name] Metric
-------------------------------------------------------------------------------
7.7.7.1/32 Remote BGP 00h06m59s 170
10.10.1.2 0
7.7.7.2/32 Remote BGP 00h06m59s 170
10.10.1.2 0
7.7.7.3/32 Remote BGP 00h06m59s 170
10.10.1.2 0
8.8.8.4/32 Remote BGP 00h06m59s 170
10.10.2.3 0
8.8.8.5/32 Remote BGP 00h06m59s 170
10.10.2.3 0
8.8.8.10/32 Remote BGP 00h06m59s 170
10.10.2.3 0
--------------------------------------------------------------------------------------
˓→-------------------------------------------

Commands to see the advertised (from VSC) and received (by uplink router) BGP IPv4 routes:

*A:Dut-G# show router bgp neighbor 10.20.1.9 advertised-routes ipv4

===============================================================================
BGP Router ID:10.20.1.7 AS:1000 Local AS:1000
===============================================================================
Legend -
Status codes : u - used, s - suppressed, h - history, d - decayed, * - valid
Origin codes : i - IGP, e - EGP, ? - incomplete, > - best, b - backup

===============================================================================
BGP IPv4 Routes
===============================================================================
Flag Network LocalPref MED
Nexthop Path-Id VPNLabel
As-Path
-------------------------------------------------------------------------------
? 7.7.7.1/32 100 0
10.15.1.254 None -
No As-Path

5.3. Port Address Translation 275

 VNS User Guide, Release 5.3.2

? 7.7.7.2/32 100 0
10.15.1.254 None -
No As-Path
? 7.7.7.3/32 100 0
10.15.1.254 None -
No As-Path
? 8.8.8.4/32 100 0
10.18.1.254 None -
No As-Path
? 8.8.8.5/32 100 0
10.18.1.254 None -
No As-Path
? 8.8.8.10/32 100 0
10.18.1.254 None -
No As-Path
-------------------------------------------------------------------------------
Routes : 6
===============================================================================

*A:Dut-G# show router bgp neighbor 10.20.1.9 advertised-routes ipv4

===============================================================================
BGP Router ID:10.20.1.7 AS:1000 Local AS:1000
===============================================================================
Legend -
Status codes : u - used, s - suppressed, h - history, d - decayed, * - valid
Origin codes : i - IGP, e - EGP, ? - incomplete, > - best, b - backup

===============================================================================
BGP IPv4 Routes
===============================================================================
Flag Network LocalPref MED
Nexthop Path-Id VPNLabel
As-Path
-------------------------------------------------------------------------------
? 7.7.7.1/32 100 0
10.15.1.254 None -
No As-Path
? 7.7.7.2/32 100 0
10.15.1.254 None -
No As-Path
? 7.7.7.3/32 100 0
10.15.1.254 None -
No As-Path
? 8.8.8.4/32 100 0
10.18.1.254 None -
No As-Path
? 8.8.8.5/32 100 0
10.18.1.254 None -
No As-Path
? 8.8.8.10/32 100 0
10.18.1.254 None -
No As-Path
-------------------------------------------------------------------------------
Routes : 6
===============================================================================

5.3. Port Address Translation 276

 VNS User Guide, Release 5.3.2

5.4 PAT to Overlay

• Overview (page 277)

• Terminology (page 278)
• Routing Policies (page 278)
– Configuration Objects (page 279)
– Preference Values (page 279)
• PAT to Overlay Configuration (page 279)
– Link Domains (page 280)
– Create Address Translation Pools (page 282)
– Define 1:1 NAT Maps (page 283)
– Add Static Route (page 283)
– Configure a Routing Policy (page 284)
• CLI Show Command Sample (page 285)

5.4.1 Overview

The PAT to Overlay feature enables the NSG with a distributed destination based PAT function from a branch domain
to a local or remote destination domain. The destination domain can be shared by multiple branch or source domains.
The requirement arises when domains with overlapping IP addressing need to communicate. For example when
multiple source domains require access to a destination domain. A destination domain subnet can be local to the
branch or remote.
Use Case Scenario 1: The NSG is deployed in a shared office environment, where multiple domains need access to a
shared resource such as a printer or local web server.

Fig. 5.26: Scenario - PAT to a Local Shared Domain

Domains DOM_A and DOM_B have subnets with overlapping IP addresses on the same NSG. Packets from DOM_A
and Dom_B destined to a DOM_Shared specific route will be PAT’ed into DOM_Shared
Use Case Scenario 2: Is a variation of Scenario 1 where the destination domain DOM_Shared is in a remote, say DC
location, and hosts ERP, NTP, NFS type functions.

5.4. PAT to Overlay 277

 VNS User Guide, Release 5.3.2

Fig. 5.27: Scenario - PAT to a Remote Shared Domain

Use Case Scenario 3: Is a case for 1:1 NAT capability in the overlay. The 1:1 NAT mapping allows a branch host
to be accessed from the shared domain. For example this enables a B2B services model. This typically requires two
or more existing business networks with potentially overlapping IP addresses to be integrated without changing the IP
addressing.
The feature utilizes Domain Linking capability.

Note: In the case where the shared domain is connected to a core nextwork via BGP, PAT pools are not advertised to
BGP neighbors on the uplinks of NSGs in the shared domain. Only locally learned BGP routes are advertised to BGP
neighbors on the uplink.

5.4.2 Terminology

• Destination Domain is the outside NAT domain, owning the PAT pool. It is the shared domain to which other
domains initiate connections.
• Source Domain is the inside NAT domain. It is the domain from which connections are initiatied to the shared
domain.

Note: Terminology, such as PAT pools, 1:1 NAT, NAT Map, from the Port Address Translation (page 255) also
applies to this feature.

5.4.3 Routing Policies

You can configure a 1:1 NAT pool for a PAT to Overlay link to allow you to assign specific source IP addresses to
specific destination IP addresses. However, this can result in scenarios where a source IP address is reachable by more
than one NSG. For example, two or more NSGs can learn the source IP address for an access side BGP neighbor. This
results in indeterministic route selection where the VSC selects routes based on the lowest data path ID and the route
reflector.
It is recommended that you associate a routing policy to a PAT to Overlay link to prevent indeterministic routing in
situations like the one described above. A routing policy allows you to assign next hop preferences to groups of NSGs
for all 1:1 NAT maps in a 1:1 NAT pool. For example, a routing policy could specify that all data center traffic always
prefers a particular NSG as a next hop.
Routing policies for PAT to Overlay links are only applicable for 1:1 NAT mappings. PAT IP addresses are always
unique, so there is no need to use a routing policy to resolve address duplication.

5.4. PAT to Overlay 278

 VNS User Guide, Release 5.3.2

Configuration Objects

Policy Object Group: Used to assign NSGs to a group for routing policy entries. Most use cases would require only
one NSG for a routing policy entry, but policy object groups allow you to assign multiple NSGs to an entry.
Policy Statement: Assigned to a PAT to Overlay link. You can create only one policy statement per domain link.
Routing Policy Entry: Assigns a next hop preference to a policy object group and associated 1:1 NAT pool.

Preference Values

For each routing policy entry, you can specify the Next Hop Priority in the range of 1-8. Each priority value in the
VSD correlates to a preference and local preference value for both a primary and secondary VSC.
Local preference values are used for iBGP sessions only. The value is checked only for traffic from within the same
AS. If both the source and destination NSG are associated with the same VSC, the preference value is used. If both
NSGs are associated with different VSCs, the local preference value is used.

Note: For the preference, a lower value is more preferred than a higher value. For the local preference, a higher value
is more preferred than a lower vaue.

The following table lists the VSC preference and local preference values associated with each VSD preference value.

Table 5.1: VSC preference values

Primary VSC Secondary VSC
Next Hop Priority Local Preference Preference Local Preference Preference
1 850 231 900 239
2 750 232 800 240
3 650 233 700 241
4 550 234 600 242
5 450 235 500 243
6 350 236 400 244
7 250 237 300 245
8 150 238 200 246

5.4.4 PAT to Overlay Configuration

To enable PAT to a destination domain, the branch or source domain is linked to the destination domain using a PAT
pool. The PAT pool addressing conforms to the destination domain IP address space.
The configuration of this functionality is available to the Enterprise. Once all the domains have been instantiated, the
enterprise administrator can configure PAT to Overlay.
The end-to-end configuration workflow is shown in the figure below, and involves the following steps:
• Link source domain to a destination domain
• Create Address Translation Pools
• Define 1:1 NAT Maps
• Add Static Route of type Overlay Address Translation
• Configure a Routing Policy, if required

5.4. PAT to Overlay 279

 VNS User Guide, Release 5.3.2

Fig. 5.28: End-to-end Configuration Workflow

The detailed workflow for each configuration step is described below.

Link Domains

1. Navigate to the source domain instance from Networks -> Layer 3 Domains -> My L3 Domains.
2. Select the Linking tab.

3. In the Links panel, select icon to define the domain linking between the source domain and a destination
domain.

5.4. PAT to Overlay 280

 VNS User Guide, Release 5.3.2

Fig. 5.29: Create Linked Domains

4. In the popup for New Link, select Type as Overlay Address Translation.

5. The source domain will be pre-populated. For the destination domain, click on to select a domain from the
list of domains that popup. This should be the destination domain to which the source domain must be linked.

5.4. PAT to Overlay 281

 VNS User Guide, Release 5.3.2

Create Address Translation Pools

Fig. 5.30: Create Address Translation Pool

1. Select the created link, and in the Address Translation Pools panel, click on to create a new address pool.
2. In the popup, enter the Name, Description, and address pool. The address pool is specified as a range using a
start address (First IP) (e.g. 135.10.0.24) and an end address (Last IP) (e.g. 135.10.0.34). The entire range,
inclusive of First:Last IP addresses will be assigned to NSGs that have vPorts in the source domain.

Note:
• IP addresses in the PAT pool cannot overlap with subnets in the destination domain.
• The number of IP address in the PAT pool needs to be equal or greater than the number of NSGs in the source
domain, as well as 1:1 NAT Maps.

5.4. PAT to Overlay 282

 VNS User Guide, Release 5.3.2

Define 1:1 NAT Maps

Fig. 5.31: Define 1:1 NAT Map

1. Select an address pool and click on in the Address Maps panel of the screen which will popup a New 1:1
NAT Entry screen. Configure the static 1:1 IP binding using IP addresses within the IP range of the PAT pool.

Note: The panel will also display (as read-only) dynamic PAT IPs which get auto assigned when a vPort in the source
domain is resolved on a NSG.

Add Static Route

Fig. 5.32: Add Static Route

1. For the selected domain, select the Static Route icon from the top right configuration actions.
2. Specify the Network (destination) prefix and select the route Type as Overlay Address Translation.

5.4. PAT to Overlay 283

 VNS User Guide, Release 5.3.2

Configure a Routing Policy

If you defined a 1:1 NAT pool, you may choose to configure a routing policy to prevent indeterministic routing when
source IP addresses are reachable by more than one NSG. You must create a policy object group with NSGs before
creating the routing policy. You can create only one routing policy per domain link.
Create a Policy Object Group
1. From the Networks tab, click on the Policy Object Groups button.

2. Click on the button and configure the Name and Description parameters. Click Create.

3. Select the policy from the Policy Object Groups panel and click on the button.
4. Select an NSG to add to the Policy Object Group and click Select.
5. Repeat steps 3 and 4 to add additional NSGs to the Policy Object Group, if required.
Create a Policy Statement and Policy Entries
6. From the Networks tab, click on the L3 Domains button and select an L3 domain.
7. Click on the Linking tab and select a domain link of type Overlay Address Translation.

8. Click on the Policy Statements tab and click on the button.

9. Configure the Name and Description parameters and click Create.

10. Click on the button in the Policy Entries panel.

5.4. PAT to Overlay 284

 VNS User Guide, Release 5.3.2

11. Configure the Name, Description, and Next Hop Priority parameters.
12. Select a 1:1 NAT pool and policy object group and click Create.
13. Repeat steps 10 to 12 to create more policy entries, if required.

5.4.5 CLI Show Command Sample

*A:vsc# show vswitch-controller domain-linking

===============================================================================
Domain Linking Information
===============================================================================
Src Domain Src Domain Name Dst Domain Dst Domain Name Link Type
SvcId SvcId
-------------------------------------------------------------------------------
1720836511 ncpeDomain3 1444815876 ncpeDomain1 PAT
-------------------------------------------------------------------------------
No. of domain links: 1
===============================================================================

To list the 1:1 NAT Maps on an NSG, run the following whitelisted command using the destination domain service
ID.
[nuage@nsg ~]# ovs-appctl vrf/nat-routes 1444815876 alubr0
---------+-----------------+-----------------+----------+--------+------------+-------
˓→------

IP | Nat-IP | Nat-vrf | Duration | Cookie | Pkt Count | Pkt

˓→Bytes |

---------+-----------------+-----------------+----------+--------+------------+-------
˓→------

5.4. PAT to Overlay 285

 VNS User Guide, Release 5.3.2

20.0.0.2 | 80.0.0.72 | 0x6691e19f | 1333s | 0x1c | 0 |

˓→ 0 |
---------+-----------------+-----------------+----------+--------+------------+-------
˓→------

5.4. PAT to Overlay 286

 VNS User Guide, Release 5.3.2

5.5 Bi-Directional NAT

• Overview (page 287)

– Provider to Customer SPAT/SNAT (page 287)
– Provider to Customer DNAT (page 288)

* Provider to Customer SPAT + DNAT (page 289)

– Customer to Provider SPAT/SNAT (page 289)
• Configuration Objects (page 290)
– Provider Domain Configuration (page 290)
– Customer Domain Configuration (page 290)
– Domain Links (page 290)
• Bi-Directional NAT Processing (page 290)
• Provisioning Workflow (page 291)
– Specify SPAT Source Pool (page 291)
– Create Bi-Directional Link (page 292)
– Provider to Customer Configuration (page 292)
– Customer to Provider Configuration (page 293)
• Sample CLI Output (page 293)

5.5.1 Overview

Bi-directional NAT enables performing source NAT (SNAT) and destination NAT (DNAT) on a packet simultaneously
between a customer and a provider domain. The primary use case for this feature is in extranet scenarios where
addressing might overlap on both sides of the network or to anchor traffic to a specific link/device.
Topology assumes that many Customer domains (spoke) link to a Provider domain (hub):
• A Customer domain will only have one customer site (1 NSG or 1 NSG-RG)
• Provider domain may have multiple Provider sites
The feature enables:
• Provider to Customer SPAT/SNAT (page 287)
• Provider to Customer DNAT (page 288)
• Customer to Provider SPAT/SNAT (page 289)

Provider to Customer SPAT/SNAT

Hides Provider topology in unique space at Customer side, using a single public address per service (allowing ac-
tive/standby failover in Provider DCs)

5.5. Bi-Directional NAT 287

 VNS User Guide, Release 5.3.2

Fig. 5.33: Provider to Customer SPAT/SNAT

• Allows different provider source addresses to be SPAT’d to a single address per NSG in the customer domain.
For example, source addresses 172.16.1.1 and 172.17.1.1 SPAT’d to 192.0.2.200.
• A single application in the provider domain is exposed in the customer domain via 1:1 SNAT mapping.
• SPAT mapping is only for configured prefixes in Provider domain (source list)
• SPAT address may be the same for every customer domain
• SPAT translations are applied at the customer NSG
• SPAT/SNAT only supported with DNAT to Customer domain (page 288)

Provider to Customer DNAT

Fig. 5.34: Provider to Customer DNAT

• Mapping specific Customer domain destination IP addresses 1:1 to Customer alias IP addresses (from the
Provider DNAT Pool) in the Provider Domain
• Every Customer domain will have a DNAT pool assigned by the Provider (e.g. 100.64.0.0/24, 100.64.1.0/24) to
hide customer topologies and eliminate overlapping address space between Customer and Provider domain

Note: Provider to Customer DNAT = Customer to Provider SNAT

5.5. Bi-Directional NAT 288

 VNS User Guide, Release 5.3.2

Provider to Customer SPAT + DNAT

The DNAT rule must be used in conjunction with SPAT/SNAT

Fig. 5.35: Bi-directional NAT

In the illustrated example, Provider_Source = 172.16.1.1 and Provider_Destination = 100.64.0.1 translates to Cus-
tomer_Source = 192.0.2.200 and Customer_Destination = 192.168.22.1

Customer to Provider SPAT/SNAT

Fig. 5.36: Customer to Provider SPAT/SNAT

• SNAT (1:1) if 1:1 DNAT rule exists in the reverse direction

• SPAT (N:1) whereby Customer source IP address(es) are SPAT’d to an address from assigned pool in Provider
domain. This is the PAT to Overlay use case with native IP route in Provider domain. Requires static route
type Overlay Address Translation to be configured.
• Customer source addresses may be directly attached subnets or may be BGP learned subnets

Note:
• SPAT/SNAT pool (Provider alias) must not overlap with Customer domain routes
• SPAT/SNAT routes are installed (and routable) in each customer domain.

5.5. Bi-Directional NAT 289

 VNS User Guide, Release 5.3.2

• Static route from customer domain to provider domain is required. Customer domain routes are not leaked back
into Provider domain.

5.5.2 Configuration Objects

Provider Domain Configuration

• SPAT source pool configured per customer. A pool identifies a group of applications in the Provider domain.
• A collection of Provider domain IP ranges which will serve as alias IPs of Customer domain IPs.
• SPAT IPs (reserved from the Provider domain IP ranges). There should be at least one SPAT IP per NSG in the
customer domain.
• Provider to Customer Translation Map:
– 1:1 NAT mapping of Customer domain IP to the alias IP in the Provider domain.
– SPAT mapping, whereby mapping of a SPAT source pool to a SPAT IP gets dynamically assigned to an
NSG (Datapath ID). The SPAT IP to NSG assignment cannot be statically configured.

Customer Domain Configuration

• A collection of customer domain IP ranges which will serve as alias IPs of Provider domain IPs.
• Customer to Provider Translation Map:
– 1:1 NAT mapping of Provider domain IP to the alias IP in the customer domain.
– Auto-created N:1 PAT mapping of NSG Datapath ID to a alias IP in the customer domain. In the case of
NSG-RG, two PAT maps are auto-created (one for each NSG)
• As in the PAT to Overlay case, a static route (of type Overlay Address Translation) from customer domain to
provider domain is required.

Domain Links

A new link type called Bi-Directional NAT is required as part of the configuration. The destination (Customer) domain
owns the links.

5.5.3 Bi-Directional NAT Processing

For traffic from Provider domain to Customer domain:

DNAT
• If packet destination IP match is found in 1:1 Provider to Customer Translations, then the corresponding Cus-
tomer IP is used for DNAT in the customer domain.
• If no matching entry is found, then packet destination is unchanged.
SNAT/SPAT
• If packet source IP match is found in 1:1 Provider to Customer Translations, then corresponding alias IP is used
for SNAT in the customer domain. Else,

5.5. Bi-Directional NAT 290

 VNS User Guide, Release 5.3.2

– If packet source IP match is found in N:1 Provider to Customer Translations, then corresponding SPAT IP
is used for SNAT in the customer domain.
• If no matching entry is found, then packet source is unchanged.
For traffic from customer domain to provider domain:
DNAT
• If packet destination IP match is found in 1:1 Customer to Provider Translations, then the corresponding Provider
IP is used for DNAT in the provider domain.
• If there is a matching entry, traffic is forwarded as PAT to Overlay using static route.
SNAT
• If packet source IP match is found in 1:1 Customer to Provider Translations, then corresponding alias IP is used
for SNAT in the provider domain.
• If no matching entry is found, then packet source is unchanged.

5.5.4 Provisioning Workflow

The workflow is in the context of CSPRoot.

Specify SPAT Source Pool

Fig. 5.37: Add SPAT Source IP Addresses

1. Navigate to Provider domain instance and select the IP configuration icon from the rightmost panel to
specify the SPAT Source Pool (which is the source IPs of applications for SPAT).
2. Enter a pool name and a list of IP addresses.

5.5. Bi-Directional NAT 291

 VNS User Guide, Release 5.3.2

Create Bi-Directional Link

Fig. 5.38: Add Bi-Directional Link

1. Navigate to Customer domain instance and select Linking tab.

2. In the Links panel, add a link of Type Bi-Directional NAT. For Source, select a Provider domain.

Provider to Customer Configuration

Fig. 5.39: Provider to Customer Configuration

1. For the created Bi-Directional link, select the Provider to Customer tab, and create Provider Source Pool
(which are the alias IPs of Customer domain IPs).
2. For the selected Provider Source Pool, select Provider to Customer Translation Map tab and add 1:1 NAT
maps (Customer IPs to Alias IPs).

5.5. Bi-Directional NAT 292

 VNS User Guide, Release 5.3.2

Fig. 5.40: Provider SPAT Configuration

3. Select Provider SPAT tab to specify the SPAT IP (from the Provider Source Pool) for the reserved SPAT source
pool. The PAT map will be displayed in the Provider to Customer Translation Map tab.

Customer to Provider Configuration

Fig. 5.41: Customer to Provider Configuration

1. For the created Bi-Directional link, select the Customer to Provider tab, and create Customer Source Pool
(which are the alias IPs of Provider domain IPs).
2. Select Customer to Provider Translation Map to specify 1:1 NAT maps (provider IPs to alias IPs)

Note: When Customer Source Pool is created, PAT IP is auto-created and dynamically assigned to customer domain
NSG. Dynamically assigned PAT IP will always use datapath ID.

5.5.5 Sample CLI Output

*A:vsc# show vswitch-controller domain-linking detail

===============================================================================
Domain Linking Information Detail
===============================================================================

5.5. Bi-Directional NAT 293

 VNS User Guide, Release 5.3.2

Src Dom SvcId : 681418080 Src Dom Id : 39947

Src Domain name : VisaDomain
Src Dom Ent name : test_organization
Src Tun Type : VXLAN
Src Dom GRT Leak : disabled Src Dom Leakable : False
Adv Criteria : none
Dest Dom SvcId : 536305307 Dest Dom Id : 988848
Dest Dom Name : CustDomain1
Dest Dom Ent Name : test_organization
Dest Dom Tun Type : VXLAN
Dest Dom GRT Leak : disabled Dest Dom Leakable : False
Link Type : bidir

------------------------------------------------------------------------------
Bidir Mapping Information
------------------------------------------------------------------------------
Pool Uuid Ip/Nsg-Id Alias/Spat-Ip Map Type
------------------------------------------------------------------------------
d4896404-a9b4-46cb-b71d-4ba0d3566339 30.0.0.2 200.0.0.2 c-nat
30.0.0.3 200.0.0.24 c-nat
193.6.26.77 200.0.0.77 c-spat
f8339e0d-3686-4266-a1f8-0d114ca5ea39 20.0.0.2 100.0.0.10 p-nat
193.6.26.77 100.0.0.60 p-spat
------------------------------------------------------------------------------
No. of Bidir Mappings: 5
------------------------------------------------------------------------------

-------------------------------------------------------------------------------
No. of domain links: 1
===============================================================================

5.5. Bi-Directional NAT 294

 VNS User Guide, Release 5.3.2

5.6 Service Chaining in VNS

• Overview (page 295)

– 1. RT in DC domain via NSG-BR using BR connection (page 295)
– 2. RT on Branch vPort in a disjoint underlay via NSG-UBR (page 296)
– 3. RT on NSG-BR/NSG-UBR vPort in common underlay (page 296)
• Service Chaining Configuration (page 296)
– Define Redirect Target (RT) (page 297)
– Associate RT to a vPort (page 297)
– Define Forwarding Policy in Branch Domain (page 298)
• Sample CLI Output (page 298)
– Example Output for Use Case 1 (page 298)
– Example Output for Use Case 3 (page 301)

5.6.1 Overview

Using advanced forwarding policy rules, an enterprise administrator can apply service chaining rules to one or more
hosts in the service, and redirect all or specific application flows, from a branch domain, to a L3 Redirection Target
(RT). Return path from redirection target towards actual destination (in branch domain) will follow existing behavior.
Following use cases are supported to enable service chaining from NSG in branch site to:
• A linked DC domain, where linking is enabled by an NSG-BR (Use Case 1 (page 295))
• Another branch NSG in a deployment with disjoint underlays, where the datapath between the sites is enabled
via the NSG-UBR (Use Case 2 (page 296))
• NSG-BR or NSG-UBR vPort in common underlay (Use Case 3 (page 296))
The corresponding Service Chaining configuration (page 296) workflow (which is common to all use cases) is de-
scribed later in this chapter.

1. RT in DC domain via NSG-BR using BR connection

Fig. 5.42: Service Chaining Use Case 1

5.6. Service Chaining in VNS 295

 VNS User Guide, Release 5.3.2

Note: ECMP is not supported for redirection targets in DC domain when there are multiple BR links from an NSG.
In such a scenario, one NSG-BR will be selected using an internal heuristic and the selection is not predictable.

Service chaining is also supported across enterprises, such as branch domain in Enterprise 1 and L3 redirection target
in a DC domain in Enterprise 2. Service chaining to a redirection target of type virtual wire is not supported.

2. RT on Branch vPort in a disjoint underlay via NSG-UBR

Fig. 5.43: Service Chaining Use Case 2

3. RT on NSG-BR/NSG-UBR vPort in common underlay

In this scenario, traffic will traverse underlay uplinks (in common underlay). The vPort on the NSG-BR or NSG-UBR
is in the same domain as the branch NSG vPort. The NSG-UBR use case is depicted below:

Fig. 5.44: Service Chaining Use Case 3

Note: For use case 3, NSG Group to UBR Group Bindings (page 173) must also be configured in addition to the
service chaining configuration.

5.6.2 Service Chaining Configuration

The workflow for defining the advanced forwarding policy to a RT is as follows:

5.6. Service Chaining in VNS 296

 VNS User Guide, Release 5.3.2

Define Redirect Target (RT)

Fig. 5.45: Define Redirection Target

1. As Enterprise administrator, navigate to Networks -> Layer 3 Domains, and select a Domain instance in which
the RT is to be defined.

2. Select the Policies tab, and select Redirection Target .

3. In the Redirection Target panel, select the at the bottom of the panel to define a new RT.
4. In the Redirection Target popup, enter Name, Description, and Service Insertion Type as L3. Select the Allow
Redundant Appliances checkbox, if more than one VPort is to be specified.
5. Select Create to define the RT.

Associate RT to a vPort

1. Select the RT, and in the VPorts panel, select an existing destination vPort per the use case being configured.
A Virtual IP will be automatically created. If redundancy is enabled on the RT, then two VPorts may be associated.

Note: Unlinking a VPort does not delete the associated Virtual IP. It must be explicitly deleted.

5.6. Service Chaining in VNS 297

 VNS User Guide, Release 5.3.2

Define Forwarding Policy in Branch Domain

Fig. 5.46: Define Forwarding Policy

1. As Enterprise administrator, navigate to Networks -> Layer 3 Domains, and select a Branch Domain instance
for which the forwarding policy is to be defined.

2. Select the Policies tab, and select Forwarding Policies .

3. In the Forwarding Policies panel, select the at the bottom of the panel to define a new advanced policy.
4. In the Forwarding Policy Entry popup, enter Name, Priority and application flow match rules.
5. Select ACL Type as Local.
6. Enter the Origin Location and Destination Network. If a policy group is desired, then it should be defined
separately (not shown in this workflow).
7. If the RT is in a Linked domain, then Select enterprise (if different) and Select domain of the redirection target.
Otherwise skip this step.
8. Select Action as Redirect. Link to the created redirection target.
9. Select Create/Update to create/update the policy.

5.6.3 Sample CLI Output

Example Output for Use Case 1

*A:vsc# show vswitch-controller esi-routes enterprise "test_organization_1" domain

˓→"dcDomain1"

===============================================================================
ESI Routes

5.6. Service Chaining in VNS 298

 VNS User Guide, Release 5.3.2

===============================================================================

===============================================================================
Legend:
Flag : P -> Primary, S -> Secondary, V -> Virtual Next Hop on NAT, I -> IPSEC
===============================================================================
Flag ESI NextHop Owner
-------------------------------------------------------------------------------
--- ff:ff:ff:00:00:00:00:00:27:11 10.15.34.254 BGP_VPN
-------------------------------------------------------------------------------
No. of ESI routes: 1
===============================================================================
===============================================================================

*A:vsc# show vswitch-controller esi-routes svcId 1246362415

===============================================================================
ESI Routes
===============================================================================

===============================================================================
Legend:
Flag : P -> Primary, S -> Secondary, V -> Virtual Next Hop on NAT, I -> IPSEC
===============================================================================
Flag ESI NextHop Owner
-------------------------------------------------------------------------------
--- ff:ff:ff:00:00:00:00:00:27:11 10.15.34.254 BGP_VPN
-------------------------------------------------------------------------------
No. of ESI routes: 1
===============================================================================
===============================================================================

*A:vsc# show vswitch-controller esi-routes enterprise "test_organization_1" domain

˓→"ncpeDomain1"

===============================================================================
ESI Routes
===============================================================================

===============================================================================
Legend:
Flag : P -> Primary, S -> Secondary, V -> Virtual Next Hop on NAT, I -> IPSEC
===============================================================================
Flag ESI NextHop Owner
-------------------------------------------------------------------------------
-V- ff:ff:ff:00:00:00:00:00:27:11 10.15.1.254 NVC_LEAK
-------------------------------------------------------------------------------
No. of ESI routes: 1
===============================================================================

*A:vsc# show vswitch-controller esi-routes svcId 1089170746

===============================================================================
ESI Routes
===============================================================================

===============================================================================

5.6. Service Chaining in VNS 299

 VNS User Guide, Release 5.3.2

Legend:
Flag : P -> Primary, S -> Secondary, V -> Virtual Next Hop on NAT, I -> IPSEC
===============================================================================
Flag ESI NextHop Owner
-------------------------------------------------------------------------------
-V- ff:ff:ff:00:00:00:00:00:27:11 10.15.1.254 NVC_LEAK
-------------------------------------------------------------------------------
No. of ESI routes: 1
===============================================================================
===============================================================================

*A:vsc# show vswitch-controller esi-routes enterprise "test_organization_2" domain

˓→"ncpeDomain2"

===============================================================================
ESI Routes
===============================================================================

===============================================================================
Legend:
Flag : P -> Primary, S -> Secondary, V -> Virtual Next Hop on NAT, I -> IPSEC
===============================================================================
Flag ESI NextHop Owner
-------------------------------------------------------------------------------
-V- ff:ff:ff:00:00:00:00:00:27:11 10.15.1.254 NVC_LEAK
-------------------------------------------------------------------------------
No. of ESI routes: 1
===============================================================================
===============================================================================

*A:vsc# show vswitch-controller esi-routes svcId 144580012

===============================================================================
ESI Routes
===============================================================================

===============================================================================
Legend:
Flag : P -> Primary, S -> Secondary, V -> Virtual Next Hop on NAT, I -> IPSEC
===============================================================================
Flag ESI NextHop Owner
-------------------------------------------------------------------------------
-V- ff:ff:ff:00:00:00:00:00:27:11 10.15.1.254 NVC_LEAK
-------------------------------------------------------------------------------
No. of ESI routes: 1
===============================================================================

Following output is on the NSG-BR

# ovs-appctl lport/show alubr0
lportid ffff:ff00:0000:0000:2711, gen_id 3, mode l3, vrf_id 1089170746
˓→evpn_id 0

vni_id 5546154, gen_id 3, tep_name lpx-c4ef6933, vrf_id 1246362415,

˓→local_tep_ip 166.220.62.206

vni_id 5546154, gen_id 3, tep_name lpx-e353e34b, vrf_id 1246362415,

˓→local_tep_ip 10.18.1.254

vni_id 5546154, gen_id 3, tep_name lpx-11308804, vrf_id 1246362415,

˓→local_tep_ip 10.15.1.254

5.6. Service Chaining in VNS 300

 VNS User Guide, Release 5.3.2

lportid ffff:ff00:0000:0000:2711, gen_id 3, mode l3, vrf_id 144580012

˓→ evpn_id 0
vni_id 5546154, gen_id 3, tep_name lpx-34d816ae, vrf_id 1246362415,
˓→local_tep_ip 166.220.62.206
vni_id 5546154, gen_id 3, tep_name lpx-27ea38a9, vrf_id 1246362415,
˓→local_tep_ip 10.18.1.254

vni_id 5546154, gen_id 3, tep_name lpx-f9025595, vrf_id 1246362415,

˓→local_tep_ip 10.15.1.254

lportid ffff:ff00:0000:0000:2711, gen_id 3, mode l3, vrf_id 1246362415

˓→evpn_id 0

vni_id 5546154, gen_id 3, tep_name lpr-b091c0cc, tep_addr 10.15.34.

˓→254, is_virtual 0

Following output is on the NSG

# ovs-appctl lport/show alubr0
lportid ffff:ff00:0000:0000:2711, gen_id 3, mode l3, vrf_id 1089170746
˓→evpn_id 0

vni_id 5546154, gen_id 3, tep_name lpr-814bb493, tep_addr 166.220.62.

˓→206, is_virtual 1

Following output is on the VRS

# ovs-appctl lport/show alubr0
lportid ffff:ff00:0000:0000:2711, gen_id 2, mode l3, vrf_id 1246362415
˓→evpn_id 0

vni_id 5546154, gen_id 2, tep_name lpl-9f65b63d, vm_uuid cdb19e5a-

˓→b5c1-4115-b4d1-187b34cb2946, port 001658254683, vmac 00:16:58:25:46:83, resolved_

˓→to=5, local_tep_ip 10.15.34.254

Example Output for Use Case 3

*A:vsc# show vswitch-controller esi-routes enterprise "test_organization" domain

˓→"ncpeDomain1"

===============================================================================
ESI Routes
===============================================================================

===============================================================================
Legend:
Flag : P -> Primary, S -> Secondary, V -> Virtual Next Hop on NAT, I -> IPSEC
===============================================================================
Flag ESI NextHop Owner
-------------------------------------------------------------------------------
-V- ff:ff:ff:00:00:00:00:00:27:10 va-10.15.2.254/1/4 NVC
-------------------------------------------------------------------------------
No. of ESI routes: 1
===============================================================================

Output on NSG which has the RT

# ovs-appctl lport/show alubr0
lportid ffff:ff00:0000:0000:2710, gen_id 1, mode l3, vrf_id 1139714140
˓→evpn_id 0

vni_id 1760067, gen_id 1, tep_name lpl-a9920654, vm_uuid e9ae425b-2e47-

˓→467c-a62d-105c865d8f30, port port4.121, vmac 00:10:34:42:17:01, resolved_to=31,

˓→local_tep_ip 74.191.36.199

5.6. Service Chaining in VNS 301

 VNS User Guide, Release 5.3.2

vni_id 1760067, gen_id 1, tep_name lpl-4602941e, vm_uuid e9ae425b-2e47-

˓→467c-a62d-105c865d8f30, port port4.121, vmac 00:10:34:42:17:01, resolved_to=31,
˓→local_tep_ip 10.18.2.254

vni_id 1760067, gen_id 1, tep_name lpl-08862ce2, vm_uuid e9ae425b-2e47-

˓→467c-a62d-105c865d8f30, port port4.121, vmac 00:10:34:42:17:01, resolved_to=31,
˓→local_tep_ip 10.15.2.254

Output on other NSG with same/different underlay as the NSG with redirect targert:

# ovs-appctl lport/show alubr0

lportid ffff:ff00:0000:0000:2710, gen_id 1, mode l3, vrf_id 1139714140
˓→evpn_id 0

vni_id 1760067, gen_id 1, tep_name lpr-360d6b4f, tep_addr 74.191.36.

˓→199, is_virtual 1

Output on NSG-UBR

# ovs-appctl lport/show alubr0

lportid ffff:ff00:0000:0000:2710, gen_id 1, mode l3, vrf_id 1139714140
˓→evpn_id 0

vni_id 1760067, gen_id 1, tep_name lpx-5bcda513, vrf_id 1139714140,

˓→local_tep_ip 75.140.61.47

vni_id 1760067, gen_id 1, tep_name lpx-9a70584b, vrf_id 1139714140,

˓→local_tep_ip 10.15.1.254

vni_id 1760067, gen_id 1, tep_name lpx-0d30c098, vrf_id 1139714140,

˓→local_tep_ip 11.11.11.11

vni_id 1760067, gen_id 1, tep_name lpr-a54003a3, tep_addr 74.191.36.

˓→199, is_virtual 1

vni_id 1760067, gen_id 1, tep_name lpx-e58dd9a4, vrf_id 1139714140,

˓→local_tep_ip 12.12.12.12

vni_id 1760067, gen_id 1, tep_name lpx-7ea68c1d, vrf_id 1139714140,

˓→local_tep_ip 13.13.13.13

vni_id 1760067, gen_id 1, tep_name lpx-65fe3614, vrf_id 1139714140,

˓→local_tep_ip 10.18.1.254

5.6. Service Chaining in VNS 302

 VNS User Guide, Release 5.3.2

5.7 DNA Subnets

• Overview (page 303)

– Proxy ARP (page 303)
• Sample Use Cases (page 304)
– Sample #1 (page 304)
– Sample #2 (page 304)
• Configuration (page 305)
– Disable Advertising on a Subnet (page 305)
– Configure a Proxy ARP Filter (page 305)
• CLI Samples (page 306)

5.7.1 Overview

Do Not Advertise (DNA) subnets allow you to configure a subnet with routes that are not advertised into the network
overlay or into the WAN. With limited advertising, DNA subnets allow for more flexibility with overlapping IP ad-
dresses within a domain. This allows you to use the same DNA subnet behind NSGs in different branch sites that are
all on the same domain. .. .. Local static routes configured on a DNA subnet are advertised to the LAN using a local
routing protocol such as BGP or OSPF. These static routes must be configured with the NSG as the next hop. .. ..
Secondary IP addresses can be configured outside or inside the range of the DNA subnet. The secondary IP address
must be advertised by the NSG with itself as the next hop.
Overlapping IP addresses between DNA subnets are allowed based on the following requirements:
• Overlapping IP addresses are allowed between DNA subnets in the same domain, unless the two DNA subnets
are situated behind the same NSG.
• Overlapping IP addresses are allowed between DNA subnets in different domains behind the same NSG.
• Overlapping IP addresses are not allowed between a DNA subnet and a non-DNA subnet.
• The virtual IP (VIP) must be unique for each DNA subnet.

Note: In DNA subnets, only static route and VIP endpoints are eligible to receive traffic from other subnets.

Proxy ARP

In cases where VIPs are used in a duplicated DNA subnet, you can configure a proxy ARP to blacklist used IP ranges
to ensure VIPs are not duplicated.
When ARP is enabled, ARP probes are sent at regular intervals to verify connectivity between hosts. For regular and
redirect VIPs, ARP probes are sent every 5 seconds on the V-Port until a VIP is learned. When the VIP is learned,
probes continue to be sent every 60 seconds. If the VIP is unlearned, the NSG continues probing at 5 second intervals.
As long as the next-hop IP is not learned, ARP probes are flooded on all local V-Ports into the EVPN.

5.7. DNA Subnets 303

 VNS User Guide, Release 5.3.2

5.7.2 Sample Use Cases

Sample #1

In the following example, a DNA subnet is configured and used behind multiple NSGs. The associated static routes
(VSD-configured) need to be unique across the domain and are advertised to the network overlay or by BGP to the
WAN.
Each switch has a management secondary IP address that faces the PE. Local static routes assign each secondary IP to
a next hop within the DNA subnet.

Sample #2

In the following example, a DNA subnet is configured and used behind multiple NSGs. The subnet consists of WiFi
access points that are reusing private IP address space in multiple branches. These WiFi access points are controlled
by a master access point with a VIP. The master VIP in each subnet is statically assigned, and is unique to the subnet.
It is the only IP that communicates with the management VRF outside of the subnet. The master IP address can switch
from one VIP to the other, so a proxy ARP is required to blacklist previously used VIPs and prevent any duplication
among the reused DNA subnets. The VSC installs a route to the VIP with a next-hop of the respective DNA subnet
NSG uplink IP. This allows the VIP to be the only IP advertised into the VPN MPLS.

5.7. DNA Subnets 304

 VNS User Guide, Release 5.3.2

5.7.3 Configuration

Disable Advertising on a Subnet

Perform the following steps to disable advertising to the overlay and BGP WAN on an existing subnet.

1. From the top-level menu, click on the Networks tab and click the L3 Domains button.
2. Select an L3 domain and a subnet.
3. On the Subnet tab, disable the Advertise to Overlay parameter.
4. Click Update to save the changes.

Configure a Proxy ARP Filter

Perform the following steps to enable proxy ARP on a subnet and configure a filter.

1. From the top-level menu, click on the Networks tab and click the L3 Domains button.
2. Select an L3 domain and a subnet.
3. On the Subnet tab, enable the Enable Proxy ARP parameter and click Update to save the changes.

4. Click on the Proxy ARP Filter button.

5. Click on the button.

5.7. DNA Subnets 305

 VNS User Guide, Release 5.3.2

6. Configure the Network Type, First Address, and Last Address parameters to define a range of blacklisted IP
addresses.
7. Click Create.

5.7.4 CLI Samples

Use the following command to check the list of static route next-hops.

ovs-appctl evpn/vip-list-show alubr0

Evpn Arp Vips:

evpn_id: 1569489553
EVPN ARP VIP IP: 20.0.0.3 gen_id: 15 arp_probe: true del_pending: false

evpn_id: 94500459
EVPN ARP VIP IP: 192.0.102.6 gen_id: 15 arp_probe: true del_pending: false

Use the following command to check provisioned VIPs.

ovs-appctl vm/port-vip-list-show 08eb8a5b-1e4a-4f48-8059-a789243bb645 host

Name: ovs1-port3-vlan32 UUID: 08eb8a5b-1e4a-4f48-8059-a789243bb645
Name: port3.32 MAC: 00:13:39:19:41:87
Redirect Vips:

Regular Vips:
VIP IP: 192.0.102.200 VIP MAC: 00:00:00:00:00:00
VIP IP: 192.0.102.100 VIP MAC: 00:00:00:00:00:00

FIP Vips:

Route Nexthop Vips:

5.7. DNA Subnets 306

 VNS User Guide, Release 5.3.2

5.8 QoS

• QoS overview (page 308)

– Supported QoS features (page 308)
• Ingress and egress QoS (page 308)
– Ingress and egress QoS use cases (page 309)

* Scenario 1: Network WAN-side egress shaping (page 309)

* Scenario 2: Access LAN-side egress shaping (page 310)
– QoS algorithm (page 311)

* QoS algorithm sample (page 311)

– QoS statistics (page 312)
– NSG-UBR QoS (page 312)
• Control traffic QoS (page 312)
– Control traffic QoS for the VSC (page 312)
• Ingress and egress QoS configuration specifics (page 313)
– QoS policies (page 313)
– DSCP mapping tables (page 313)
– Ingress forwarding policies - DSCP remarking (page 313)
– Egress QoS policies (page 313)
– Rate limiters (page 314)
– Remarking profiles (page 314)
– Ingress QoS policers (page 315)
– QoS statistics configuration specifics (page 315)

* Statistics panel (page 316)

• QoS workflow (page 318)
• Procedures for QoS (page 318)
– Creating rate limiters (page 318)
– Creating DSCP remarking profiles (page 318)
– Creating DSCP remarking profiles (page 319)
– Creating QoS policers (page 319)
– Creating egress QoS policies (page 319)
– Assigning QoS policies to a port/VLAN (page 320)
– Displaying egress QoS statistics (page 320)
• Control traffic QoS procedure (page 320)
• QoS CLI samples (page 321)

5.8. QoS 307

 VNS User Guide, Release 5.3.2

– Configuring remarking for control traffic on the VSC (page 322)

5.8.1 QoS overview

Nuage VNS Quality of Service (QoS) allows you to configure the rate at which traffic ingresses or egresses NSG
ports. QoS uses ingress traffic policing and egress traffic shaping to ensure that performance-sensitive applications are
prioritized, fairness is maintained, and packet loss is minimized.
Key functionality of QoS includes the following:
• DSCP mapping for up to eight forwarding classes.
• Traffic rate limiting and classification for both network and access ingress points.
• Hierarchical shaping, scheduling, and queueing per port/VLAN for both network and access egress points.
• DSCP and CoS remarking for up to eight forwarding classes.
• DSCP and CoS remarking for self-generated NSG control traffic.
As part of the workflow for this feature, VNS supports the DSCP mapping and access LAN-side ingress traffic policing
functionalities included as part of the VSP solution. DSCP mapping and access LAN-side ingress traffic policing can
be configured for VNS deployments at the domain, zone, subnet, or vPort level. See the QoS chapter in the VSP User
Guide for more information about these functionalities.

Supported QoS features

Nuage VNS currently provides support for the following QoS features on the NSG:
• Ingress and egress QoS (page 308)
• Control traffic QoS (page 312)

5.8.2 Ingress and egress QoS

Ingress and egress QoS entails the policing of ingress traffic and shaping of egress traffic from either the access LAN-
side or network WAN-side.
The following workflow describes the QoS process in the example of a packet received on the LAN side and forwarded
to the WAN.
1. Ingress rate limiting (policing) — Access LAN-side traffic ingresses a service-specific port/VLAN and is
controlled with rate limiters. Rate limiters allow you to define the maximum rate at which traffic can be received
from specific sources. You can specify separate rate limiters for unicast and BUM traffic.
2. Ingress classification — Traffic is mapped to one of eight forwarding classes based on the IPv4 DSCP or CoS
value in the customer packet.
3. DSCP remarking for customer packet — DSCP bits for the inner payload packets are modified based on
actions defined for the ingress forwarding policy entry.
4. DSCP and CoS remarking for the uplink outer tunnel header — DSCP or CoS bits for the uplink outer
tunnel header are modified based on remarking policies. Remarking policies allow you to redefine forwarding
class assignments per uplink based on a user-defined mapping table. Inner packet markings from the LAN side
are not modified.

5.8. QoS 308

 VNS User Guide, Release 5.3.2

5. Egress queueing — Traffic is assigned to four egress queues based on forwarding classes. One queue is based
on strict priority queuing and three are based on weighted round robin. Bandwidth is allocated to the four queues
based on the QoS algorithm.

Fig. 5.47: QoS packet flow

Note: In addition to the user traffic queues, there is also a network control queue reserved for VNS control traffic
(such as OpenFlow and JSON). The control traffic queue is assigned the highest priority by default. Refer to Control
traffic QoS (page 312) for more information about control traffic QoS.

Ingress and egress QoS use cases

The use cases in this section show some examples of traffic flow along with the process of ingress policing and egress
shaping.

Scenario 1: Network WAN-side egress shaping

In the following use case, the NSG is deployed at a site to provide WAN connectivity to another site using an overlay
tunnel.

Fig. 5.48: Network uplink egress shaping with overlay tunnel

In the following use case, the NSG is deployed at a site to provide local Internet breakout at a site without an overlay
tunnel.

5.8. QoS 309

 VNS User Guide, Release 5.3.2

Fig. 5.49: Network uplink egress shaping without tunnel

In either case, the network WAN-side port of the NSG can be a potential congestion point if the network uplink is
of a lower speed than the aggregate rate of the access LAN-side ports bandwidth. In such deployments, QoS enables
service-unaware traffic shaping rules at port level using the ingress DSCP values present in the customer header.
Shaping the network egress port allows the NSG to ensure that the WAN link is not congested. The feature is used as
a bandwidth-on-demand capability to apply a QoS profile for a predetermined amount of time, or to address a WAN
uplink bandwidth upgrade or downgrade.

Scenario 2: Access LAN-side egress shaping

In the following use case, QoS is used at the port/VLAN level when the cumulative bandwidth of the traffic destined to
a single access port/VLAN from a combination of network uplink ports or other LAN-side ports is greater than what
the egress port/VLAN can support.

Fig. 5.50: Access LAN-side egress shaping

In the following use case, QoS is used at the port/VLAN level when the NSG is deployed as an NSG-V at a CO or
data center and an L2 point-to-point service is extended between the NSG and the customer site. The L2 P2P service
itself is prescribed a certain bandwidth profile and consequently the NSG is required to shape the rate.

5.8. QoS 310

 VNS User Guide, Release 5.3.2

Fig. 5.51: Access LAN-side egress shaping

QoS algorithm

VNS QoS uses a Hierarchical Token Bucket (HTB) scheduler to shape customer traffic. The scheduler includes a
parent queue with four child queues. Bandwidth is allocated to the child queues based on strict priority and weighted
round robin (WRR) queueing methods. The queueing method used for each child queue does not change if the
available bandwidth falls below the CIR of the parent queue.
• Queue 1 (Q1): strict priority queue
• Queues 2-4 (Q2-4): weighted round robin queues
The QoS algorithm allows you to assign low latency and delay-sensitive traffic to the priority queue (Q1) for guaranteed
throughput without starving the lower priority queues (Q2-4) during times of network congestion.
During network congestion, bandwidth is distributed in the following order:
1. Bandwidth is allocated to Q1 up to the configured CIR value.
2. Bandwidth is distributed among Q2, Q3, and Q4 using WRR up to their configured CIR values. The CIR values
are used to determine the weight.
3. Bandwidth is allocated to Q1 up to the configured PIR value.
4. Bandwidth is distributed among Q2, Q3, and Q4 using WRR up to their configured PIR values. The CIR values
are used to determine the weight.
The HTB scheduler uses a deficit round robin (DRR) algorithm to de-queue packets from the WRR queues. The queue
length is defined by the Limit parameter in the SFQ qdisc attached to each HTB class. The default limit is 127 packets.
If there are multiple traffic flows in each queue, there is the same number of internal queues from which packets are
de-queued via round robin. The Quantum SFQ qdisc parameter defines the number of bytes de-queued from each
internal queue.

QoS algorithm sample

Q0 is the parent queue and Q1-4 are child queues with the following configured attributes:
Queue CIR PIR
Q0 1 Mbps 1.5 Mbps
Q1 400 Kbps 1.5 Mbps
Q2 200 Mbps 1.5 Mbps
Q3 300 Kbps 1.5 Mbps
Q4 100 Kbps 1.5 Mbps

5.8. QoS 311

 VNS User Guide, Release 5.3.2

Scenario 1: In the following scenario, bandwidth is first allocated to Q1 up to the configured CIR value. Then,
bandwidth is distributed among Q2, Q3, and Q4 up to their configured CIR values using WRR. The remaining 500
Kbps is allocated to Q1.
Queue Input Rate Scheduling Rate Weight
Q1 1.5 Mbps 900 Kbps n/a
Q2 1 Mbps 200 Kbps 2
Q3 500 Kbps 300 Kbps 3
Q4 200 Kbps 100 Kbps 1
Scenario 2: In the following scenario, bandwidth is first allocated to Q1 up to the configured CIR value. Then,
bandwidth is distributed among Q2, Q3, and Q4 up to their configured CIR values using WRR. Then, bandwidth is
allocated to Q1 up to the input rate. The remaining 180 Kbps is distributed among Q2, Q3, and Q4 using WRR.
Queue Input Rate Scheduling Rate Weight
Q1 720 Kbps 720 Kbps n/a
Q2 800 Kbps 260 Kbps 2
Q3 1.2 Mbps 390 Kbps 3
Q4 400 Kbps 130 Kbps 1

QoS statistics

QoS statistics collection is enabled by default when an egress QoS policy is assigned to a port/VLAN. QoS statistics
are collected and reported for each queue.
Refer to QoS statistics configuration specifics (page 315) for more information on accessing and configuring QoS
statistics collection.

NSG-UBR QoS

Egress QoS is supported on the NSG-UBR in a single-tenant use case only. This means that the ability to prioritize
traffic between two different customers is not supported. If traffic from multiple customers is routed through the
NSG-UBR, all priority queues are treated equally and all WRR queues are part of the same hierarchy.

5.8.3 Control traffic QoS

You can configure DSCP and CoS remarking for self-generated control traffic on the NSG. This includes traffic gener-
ated for OpenFlow, DTLS, BGP, JSON, IKE, DHCP, DNS, NTP, BFD and other control plane traffic. For each NSG,
you can specify a DSCP value (0-63) and CoS value (0-7).
You can set these values by configuring the Control Traffic COS and Control Traffic DSCP parameters at the NSG
instance level. Refer to Control Traffic QoS Provisioning Workflow (page 320) for more information.

Note: Configuration of DSCP and CoS remarking for self-generated traffic is available at the NSG instance level
only. You must perform a reload configuration before the changes take effect.

Control traffic QoS for the VSC

You can configure DSCP and CoS remarking for self-generated control traffic on the VSC. Refer to Configuring
remarking for control traffic on the VSC (page 322) for a sample configuration.

5.8. QoS 312

 VNS User Guide, Release 5.3.2

5.8.4 Ingress and egress QoS configuration specifics

The following VSD objects and policies are relevant to ingress and egress QoS.

QoS policies

QoS policies allow you to apply rate limiting, forwarding class mapping, and remarking policies to access LAN-side
ingress traffic. QoS policies can be created at the domain, zone, subnet, or vPort level, with the most granular level
taking precedence.
QoS policies are applicable to both the VSP and VNS solutions. Refer to the QoS chapter in the VSP User Guide for
more information about QoS policies.

DSCP mapping tables

DSCP mapping tables define a list of mappings between customer DSCP markings and VNS forwarding classes. QoS
policies reference these mapping tables. A DSCP table can have up to 63 entries. Multiple DSCPs can map to a single
forwarding class. Only a single mapping table can be referenced by a QoS policy.

Note: The configured DSCP mapping table is implicitly referenced for NSG access LAN-side traffic. It is recom-
mended that only one DSCP mapping table be configured per domain.

DSCP mapping tables are applicable to both the VSP and VNS solutions. Refer to the QoS chapter in the VSP User
Guide for more information about DSCP mapping tables.

Ingress forwarding policies - DSCP remarking

DSCP remarking for customer payload packets can be defined in an ingress forwarding policy entry. Configure the
DSCP Remarking parameter on the Action tab and specify a forwarding class to be assigned to the customer packet.

Navigate to Network -> Domain -> Policies -> Ingress Forwarding Policies -> Policy -> Policy Entries.
Refer to the “Security and Forwarding Policies” chapter of the VSP User Guide for more information and configuration
steps for ingress forwarding policies.

Egress QoS policies

Egress QoS policies allow you to apply rate limiting, forwarding class mapping, and remarking policies to access
LAN-side egress and network WAN-side egress traffic. Egress QoS policies can be applied at the port/VLAN level
only.

Navigate to Platform Configuration -> Infrastructure -> Network Service Gateways -> QoS Configuration .
The following parameters can be configured for egress QoS policies:
• Name
• Description
• Default Service Class: Specifies the default service class. Whichever queue is assigned to that service class is
the default egress queue.
• Overall Rate: Assigns an overall rate limiter applied to the entire policy.

5.8. QoS 313

 VNS User Guide, Release 5.3.2

• Service Classes and Rates: Specify forwarding class mappings and rate limiters for the priority queue and three
WRR queues.
• DSCP Remarking Profile: Assigns a DSCP remarking profile to the policy.
• CoS Remarking Profile: Assigns a CoS remarking profile to the policy.
Consider the following when configuring an egress QoS policy:
• The eight service classes must be assigned to one or more of the four queues.
• Queues that do not have assigned service classes are not used.
• Each queue with an assigned service class must have an associated rate limiter.
Consider the following when assigning rate limiters to queues in an egress QoS policy:
– The sum of the Guaranteed Information Rate for all child queues must be less than or equal to the parent CIR. –
The Peak Information Rate for each of the child queues must be less than or equal to the parent PIR. – The Burst
Size for each of the child queue is less than or equal to the parent Burst size.

Note: A minimum CIR value of 120 kbps is recommended for each queue. If a value of 0 is assigned to a queue, the
HTB algorithm may assign a minimum default value which may be unknown and result in inconsistent allocation to
the other queues.

Rate limiters

Rate limiters allow you to specify the guaranteed (or committed) and peak rates of traffic throughput for access LAN-
side and network WAN-side egress traffic. Rate limiters are assigned to egress QoS policies and their child queues to
specify the rates at which traffic each service class egresses through the NSG.

Navigate to Platform Configuration -> Infrastructure -> Network Service Gateways -> QoS Configuration
-> Rate Limiters .
The following parameters can be configured for rate limiters:
• Name
• Description
• Guaranteed Information Rate: Specifies the committed aggregate shaped rate for parent scheduler.
• Peak Information Rate: Specifies the ceiling for the aggregate shaped rate that may be sent by the parent
scheduler.
• Burst Size: Specifies the amount that can be burst at peak speed in excess of the configured rate.

Note: Guaranteed and peak information rate values are calculated based on the traffic as it appears on the wire.
When specifying these values, you must account for the different tunnelling overheads for egress WAN QoS, VxLAN,
VxLAN+IPsec, and VxLAN+IPsec+VxLAN.

Remarking profiles

Remarking profiles allow you to configure DSCP and CoS remarking for the NSG to rewrite DSCP and CoS bits in
the uplink tunnel header. These policies allow you to redefine forwarding class assignments based on mapping tables.

5.8. QoS 314

 VNS User Guide, Release 5.3.2

For each remarking policy, you can map eight forwarding classes (A-H) to a DSCP value (0-63) or CoS value (0-7).
These policies can be associated with an egress QoS policy.
The VSD automatically assigns a default egress QoS policy and DSCP remarking policy to each uplink VLAN. If the
VLAN is tagged, the VSD also assigns a default CoS policy.

Navigate to Platform Configuration -> Infrastructure -> Network Service Gateways -> QoS Configuration
-> DSCP Remarking Profiles or CoS Remarking Profiles .
The following parameters can be configured for DSCP and CoS remarking profiles:
• Name
• Description
• Forwarding Class to DSCP/CoS: Selects a forwarding class (A-H)
• Map to DSCP/CoS: Specifies the DSCP or CoS value to be mapped to the selected forwarding class.
Consider the following when configuring DSCP or CoS remarking profiles:
• You can define only eight forwarding class mappings for each policy.
• DSCP and CoS remarking profiles cannot be created with the same forwarding class.
• DSCP and CoS remarking profiles can be associated with egress QoS policies on uplink VLANS only.
• CoS remarking profiles can be associated with egress QoS policies on tagged VLANs only.

Ingress QoS policers

Ingress QoS policers allow you to specify the rate at which network WAN-side traffic ingresses on an uplink VLAN.
You can specify a packet rate and maximum burst size for traffic ingressing via the overlay, underlay, or both. The
packet rate and burst size for underlay and overlay QoS policers are independent of one another. When the policing
threshold is crossed, the VSD raises an alarm.

Navigate to Platform Configuration -> Infrastructure -> Network Service Gateways -> QoS Configuration
-> QoS Policers.
The following parameters can be configured for ingress QoS policers:
• Name
• Description
• Rate: Specifies the rate, in Mb/s, that WAN-side traffic ingresses on the assigned VLAN.
• Burst: Specifies the maximum amount, in Kb, that can burst at peak speed in excess of the configured rate.

Note: QoS policers can be configured and applied to NSG uplinks only by users with CSProot privileges. QoS
policers can be applied only at the enterprise level. They can not be applied to a VLAN template.

QoS statistics configuration specifics

QoS statistics collection is enabled by default when an Egress QoS policy is applied to a port/VLAN. QoS statistics
are collected and reported for each queue. You can view statistics for the parent queue, network control queue, and
each child queue.
QoS statistics can viewed or retrieved using the following methods:

5.8. QoS 315

 VNS User Guide, Release 5.3.2

• Viewed graphically in the Statistics panel for the selected port/VLAN.

• Retrieved using the gateway port QoS CLI command. Refer to CLI output samples (page 321) for more infor-
mation.
• Accessed from the VSD ReST API.
The following QoS statistics are collected:
• Bytes: Specifies the total number of bytes egressing from the queue.
• Packet Count: Specifies the total number of packets egressing from the queue.
• Dropped: Specifies the number of packets dropped from a queue after surpassing the configured rate limits.
The following QoS statistics are available from CLI and ReST API only:
• Requeues: Specifies the number of times a packet has been de-queued and inserted back in the same queue
position.
• Lended: Specifies the number of tokens donated by this class.
• Borrowed: Specifies the number of tokens borrowed from parent.
• Overlimits: Specifies the number of times the queueing discipline delayed a packet.

Statistics panel

The Statistics panel can be accessed in the VSD UI from the information form for any port/VLAN. From the Statistics
panel, you can select either the Bytes or Packets to generate bar-line graphs for statistics at the byte or packet level.
The Bytes view displays a bar graph of the total bytes of traffic on the port/VLAN. The Packets view displays a bar-line
graph with total packets depicted with lines and packets dropped depicted with bars.
For each graphical view, you can configure the following:
• Date and time, using the calendar and Hourly Precision parameter.
• Ingress or Egress: Narrow statistics displayed to only ingress or egress traffic.
• Parent, Network Control, and Queue *n*: Specify one or more queues to display graphically.
On access LAN-side ports, you can also configure the following:
• Enable Stats Collection: Enables or disables statistics collection on the port.
• Collection Frequency: Specifies the frequency at which statistics are are collected on the port. The default and
recommended value is 60s.

Note: The network WAN-side ports, statistics collection is enabled with a collection frequency of 60s. This cannot
be configured.

5.8. QoS 316

 VNS User Guide, Release 5.3.2

Fig. 5.52: Bytes view with Queue 1 selected

Fig. 5.53: Packets view with all queues selected

5.8. QoS 317

 VNS User Guide, Release 5.3.2

5.8.5 QoS workflow

This section provides a high-level workflow of the tasks required to configure VNS QoS.
Step 1 Configure an organization profile with Advanced QoS enabled. Refer to the QoS chapter in the
VSP User Guide for more information.
Step 2 Configure a DSCP mapping table. Refer to the QoS chapter in the VSP User Guide for more
information.
Step 3 Configure a QoS policy. Refer to the QoS chapter in the VSP User Guide for more information
Step 4 Configure rate limiters. Refer to Creating rate limiters (page 318) for more information.
Step 5 Configure an ingress forwarding policy to define DSCP customer packet remarking. Refer to the
“Security and Forwarding Policies” chapter in the VSP User Guide for more information.
Step 5 Configure a DSCP remarking profile. Refer to Creating DSCP remarking profiles (page 318) for
more information.
Step 6 Configure a DSCP remarking profile. Refer to Creating CoS remarking profiles (page 319) for
more information.
Step 7 Configure a QoS policer. Refer to Creating QoS policers (page 319) for more information.
Step 8 Configure an egress QoS policy. Refer to Creating egress QoS policies (page 319) for more
information.
Step 9 Assign QoS policers and a QoS policy to a port/VLAN. Refer to Assigning QoS policies to a
port/VLAN (page 320) for more information.
Step 10 View QoS statistics. Refer to Displaying egress QoS statistics (page 320) for more information.

5.8.6 Procedures for QoS

This section provides the detailed steps required to configure VSD objects and policies required to enable the features
described in this chapter.

Creating rate limiters

Step 1 As CSProot, navigate to Platform Configuration -> Infrastructure -> Network Service Gate-
ways -> QoS Configuration -> Rate Limiters .

Step 2 Select to create a new rate limiter.

Step 3 Enter a Name, Description, Guaranteed Information Rate, Peak Information Rate, and Burst
Size.
Step 4 Click Create.
Step 5 Repeat steps 2 to 4 to create more rate limiters, as required.

Creating DSCP remarking profiles

Step 1 As CSProot, navigate to Platform Configuration -> Infrastructure -> Network Service Gate-
ways -> QoS Configuration -> DSCP Remarking Profiles .

5.8. QoS 318

 VNS User Guide, Release 5.3.2

Step 2 Select button to create a new DSCP remarking profile.

Step 3 Enter a Name and Description parameters and click Create.

Step 4 Select the DSCP remarking profile and click in the DSCP Remarking Policy panel.
Step 5 Enter a Forwarding Class To DSCP and Map DSCP and click Create.
Step 6 Repeat steps 4 and 5 to create up to eight forwarding class mappings, as required.

Creating DSCP remarking profiles

Step 1 As CSProot, navigate to Platform Configuration -> Infrastructure -> Network Service Gate-
ways -> QoS Configuration -> CoS Remarking Profiles .

Step 2 Select button to create a new CoS remarking profile.

Step 3 Enter a Name and Description parameters and click Create.

Step 4 Select the CoS remarking profile and click in the CoS Remarking Policy panel.
Step 5 Enter a Forwarding Class To CoS and Map CoS and click Create.
Step 6 Repeat steps 4 and 5 to create up to eight forwarding class mappings, as required.

Creating QoS policers

Note: QoS policers can be configured and applied to NSG uplinks only by users with CSProot privileges. QoS
policers can be applied only at the enterprise level. They can not be applied to a VLAN template.

Step 1 As CSProot, navigate to Platform Configuration -> Infrastructure -> Network Service Gate-
ways -> QoS Configuration -> QoS Policers.

Step 2 Select button to create a new QoS Policer.

Step 3 Enter a Name, Description, Rate, and Burst.
Step 4 Click Create.

Creating egress QoS policies

Step 1 As CSProot, navigate to Platform Configuration -> Infrastructure -> Network Service Gate-
ways -> QoS Configuration .

Step 2 Select button to create a new egress QoS policy.

Step 3 On the Scheduling tab, enter a Name, Description, and Default Service Class.
Step 4 Assign a rate limiter to the Overall Rate
Step 5 For each of the four queues, assign any number of services class and assign a rate limiter.
Step 6 Click on the Remarking tab and assign DSCP and CoS remarking profiles to the QoS policy.

5.8. QoS 319

 VNS User Guide, Release 5.3.2

Step 7 Click Create.

Assigning QoS policies to a port/VLAN

Note: This procedure can be performed at the gateway template level by CSProot or at the gateway instance level
by the enterprise admin. If a gateway instance has inherited an egress QoS policy from the gateway template, the
enterprise administrator can override them by associating a new policy at the instance level. This allows individual
gateways to have their egress QoS policies amended to suit new network uplink bandwidth profiles without affecting
the configuration of other NSGs.

Note: Existing infrastructure Egress QoS policies can be edited only if there are no associated gateway port objects.
Once the infrastructure egress QoS policy is created and is associated with gateway port/VLANs, its attributes cannot
be changed until all associations are removed. It is recommended that the CSProot or enterprise admin create a new
infrastructure egress QoS policy, disassociate the old policy, and assign the new one.

Step 1 From the enterprise view, navigate Infrastructure -> Network Service Gateways -> NSG ->
Port -> VLAN.

Step 2 Select the QoS icon .

Step 3 Assign an overlay and/or underlay QoS policer, as required.
Step 4 Assign an egress QoS policy.

Displaying egress QoS statistics

Step 1 From the enterprise view, navigate Infrastructure -> Network Service Gateways -> NSG ->
Port -> VLAN.

Step 2 Select the Statistics icon .

Step 3 Select either the Bytes or Packets view.
Step 4 Configure the date, Hourly Precision, Ingress, Egress, and queue parameters to narrow the in-
formation displayed.

Note: For access ports, the collection interval is configurable at the vPort level. For network ports, the collection
interval is set to 60 seconds and is not configurable.

Step 5 Enable or disable statistics collection, if required, by checking Enable Stats Collection.
Step 6 Enter a Collection Frequency, if required.

5.8.7 Control traffic QoS procedure

Step 1 From the enterprise view, navigate Infrastructure -> Network Service Gateways .
Step 2 Right-click an NSG and choose Edit.

5.8. QoS 320

 VNS User Guide, Release 5.3.2

Step 3 Enter a a Control Traffic COS and Control Traffic DSCP to assign remarking values for self-
generated control traffic.
Step 4 Click update.
Step 5 Perform a reload configuration to finalize the changes.

5.8.8 QoS CLI samples

This section provides sample CLI command outputs relevant to QoS.

The following sample shows how to display QoS statistics on a network port:

*A:vsc1# show vswitch-controller gateway ports 192.168.131.8 port-name "port1" qos

===============================================================================
Gateway Ports Qos Information Table
===============================================================================

Gateway IP : 192.168.131.8 Gateway Port Name : port1

Port Mode : network Port VLAN ID : 0

Ing Underlay Policer Rate : 10 Ing Overlay Policer Rate : 10

Ing Underlay Policer Burst : 1 Ing Overlay Policer Burst: 1
Qos Policy Default FC : be

Egr QoS Parent Queue Information

Egress Shaping PIR(Mbps) : 10.000 Egress Shaping PBS(kb) : 25
Egress Shaping CIR(Mbps) : 7.000
Egress Shaping Svc Classes : n/a
Number of Bytes : 0 Number of Packets : 0
Num of Packets Dropped : 0 Overlimits : 0
Requeues : 0 Lended : 0
Borrowed : 0

QoS Queue ID : 1
Egress Shaping PIR(Mbps) : 10.000 Egress Shaping PBS(kb) : 25
Egress Shaping CIR(Mbps) : 3.000
Egress Shaping Svc Classes : h1 nc
Number of Bytes : 0 Number of Packets : 0
Num of Packets Dropped : 0 Overlimits : 0
Requeues : 0 Lended : 0
Borrowed : 0

QoS Queue ID : 2
Egress Shaping PIR(Mbps) : 10.000 Egress Shaping PBS(kb) : 16
Egress Shaping CIR(Mbps) : 2.000
Egress Shaping Svc Classes : h2 ef
Number of Bytes : 0 Number of Packets : 0
Num of Packets Dropped : 0 Overlimits : 0
Requeues : 0 Lended : 0
Borrowed : 0

QoS Queue ID : 3
Egress Shaping PIR(Mbps) : 10.000 Egress Shaping PBS(kb) : 20
Egress Shaping CIR(Mbps) : 1.000
Egress Shaping Svc Classes : af l1
Number of Bytes : 0 Number of Packets : 0

5.8. QoS 321

 VNS User Guide, Release 5.3.2

Num of Packets Dropped : 0 Overlimits : 0

Requeues : 0 Lended : 0
Borrowed : 0

QoS Queue ID : 4
Egress Shaping PIR(Mbps) : 10.000 Egress Shaping PBS(kb) : 22
Egress Shaping CIR(Mbps) : 1.000
Egress Shaping Svc Classes : be l2
Number of Bytes : 0 Number of Packets : 0
Num of Packets Dropped : 0 Overlimits : 0
Requeues : 0 Lended : 0
Borrowed : 0

-------------------------------------------------------------------------------
FC to DSCP remarking profiles
-------------------------------------------------------------------------------
FC Value DSCP Value
-------------------------------------------------------------------------------
ef 46
l1 18
be 0
h1 48
nc 56
l2 8
h2 34
af 10
-------------------------------------------------------------------------------

-------------------------------------------------------------------------------
FC to COS remarking profiles
-------------------------------------------------------------------------------
FC Value COS Value
-------------------------------------------------------------------------------
h2 4
ef 5
h1 6
nc 7
l1 3
be 0
l2 1
af 2

-------------------------------------------------------------------------------
No. of Ports: 1
===============================================================================

The following sample shows how to display QoS statistics on an access port:

Configuring remarking for control traffic on the VSC

You can configure DSCP and CoS remarking for self-generated control traffic on the VSC. The following example
shows DSCP remarking configured with a value of 56 for OpenFlow.

*A:vsc-G# /configure router sgt-qos

*A:vsc-G>config>router>sgt-qos#
*A:vsc-G>config>router>sgt-qos# application openflow dscp "nc2"
*A:vsc-G>config>router>sgt-qos# info

5.8. QoS 322

 VNS User Guide, Release 5.3.2

----------------------------------------------
application openflow dscp nc2
----------------------------------------------
*A:vsc-G>config>router>sgt-qos#
[no] application - Configure DSCP/Dot1p re-marking for applications
[no] dscp - Configure DSCP name to FC mapping

*A:vsc-G>config>router>sgt-qos# application
- application <dscp-app-name> dscp {<dscp-value>|<dscp-name>}
- application <dot1p-app-name> dot1p <dot1p-priority>
- no application {<dscp-app-name>|<dot1p-app-name>}

<dscp-app-name> : bgp|cflowd|dhcp|dns|dtls-ipsec|dtls-vxlan|ftp|icmp|igmp|igmp-
˓→reporter|json-rpc|l2tp|ldp|mld|msdp|ndis|ntp|openflow|ospf|pim|radius|rip|rsvp|snmp|

snmp-
˓→notification|srrp|ssh|syslog|tacplus|telnet|tftp|traceroute|vrrp|xmpp

<dscp-value> : [0..63]
<dscp-name> :
˓→none|be|ef|cp1|cp2|cp3|cp4|cp5|cp6|cp7|cp9|cs1|cs2|cs3|cs4|cs5|nc1|nc2|af11|af12|af13|af21|af22|af2

˓→cp17|cp19|cp21|cp23|cp25|cp27|cp29|cp31|cp33|cp35|cp37|cp39|cp41|cp42|cp43|cp44|cp45|cp47|cp49|cp50
cp61|cp62|cp63
<dot1p-priority> : [none|0..7]
<dot1p-app-name> : arp|isis|pppoe

*A:vsc-G>config>router>sgt-qos# exit all

5.8. QoS 323

 CHAPTER

SIX

ROUTING PROTOCOLS

• BGP (page 325)

• OSPF (page 379)

324
 VNS User Guide, Release 5.3.2

6.1 BGP

• BGP Overview (page 326)

– MD5 Authentication (page 328)
– Multi-hop (page 328)
– BGP Session Redundancy (page 328)
• Default Route From LAN (page 330)
• VIF RTM Routes (page 330)
• BGP Configuration Workflow (page 330)
– Enable BGP and configure Local Autonomous System (AS) (page 330)
– Define Routing Policies (page 332)

* Routing Policy at Enterprise Level (page 332)

* Routing Policy at Domain Level (page 333)
– Create BGP Profile (page 334)
– BGP Peering with PE/Uplink (page 336)

* Define Uplink BGP Neighbor (page 336)

– BGP Peering with CPE/Access (page 338)

* Assign BGP Profile to Domain Template (page 338)

* Define Access-side BGP Neighbor (page 338)
• XML Configuration Requirements (page 338)
• CLI Show Command Sample (page 338)
• BGP Blob (page 339)
– Sample BGP Neighbor XML Blob (page 339)

* BGP Neighbor XML tree (page 340)

* XML Conversion (NSG) (page 340)
– Sample Routing Policy XML Blob (page 341)

* Routing policy XML tree (page 343)

* XML Conversion (SROS) (page 344)
– Order of Inheritance of Default Actions (page 345)

* Default action evaluation rules (page 345)

• BGP Policy Routing Use Cases (page 345)
– Central Gateway (page 346)

* Example #1 (page 347)

* Example #2 (page 348)
* Example #3 (page 349)

6.1. BGP 325

 VNS User Guide, Release 5.3.2

* Example #4 (page 350)

– Distributed Interworking — LAN (page 351)
– Distributed Interworking — WAN (page 352)
• BGP CLI Commands (page 353)
• VIF RTM CLI Commands (page 377)

6.1.1 BGP Overview

BGPv4 control-plane is supported on NSGs to enable advertisement/learning of LAN/WAN subnets to/from the NSG.
The primary use cases are:
1. NSG connected to upstream PE advertising LAN side subnets while learning remote routes from the PE.

Fig. 6.1: Single Uplink NSG Peering to Single PE

Fig. 6.2: Dual Uplink NSG Peering to Two PEs

2. NSG is advertising/learning routes from routers (CPE) located within the LAN (same AS) or from peering
networks (different AS).

Fig. 6.3: NSG Access-side Peering to CPE

Onboard BGP runs as CE-PE/CE-CPE protocol directly on the underlay to upstream/downstream router.
When an NSG is configured with an access-side BGP peer, it creates a TLS-encrypted MP-BGP session with the
VSC(s). There can be a maximum of two MP-BGP sessions — two on a single uplink, or one on each uplink in a dual
uplink scenario. The MP-BGP session is used to advertise BGP routes learned from the downstream CPE to the VSC,
which advertises the routes to remote NSGs. The NSG programs the data path locally after learning BGP routes, and

6.1. BGP 326

 VNS User Guide, Release 5.3.2

the VSC programs the data path for remote NSGs. BGP IPv4 routes are translated to BGP EVPN routes before being
advertised to the VSC.
The NSG-VSC BGP session fate shares with NSG-VSC OpenFlow session.
BGP routes are advertised with attribute transparency. Route attributes such as AS path, local preference, MED, and
community are retained by BGP neighbors when the route is advertised. Export and import routing policies can be
used to influence BGP path selection.
• If the same prefix is advertised by two CPEs to two different NSGs, traffic to the prefix will be forwarded using
ECMP.
• If the same prefix is advertised by two CPEs to the same NSG, only the best BGP route is advertised.
• ECMP is not used towards CPEs behind the same NSG.
• If the routes have equal attributes, the route which was learned first is selected.

Note: The NSG creates an iBGP session with the VSC using AS 65500. To prevent unexpected AS path loops, avoid
using AS 65500 elsewhere.

For Redundant Groups in the context of BGP, the following statements apply:
• Access side BGP peers on RG backup ports are kept in the “shut” state.
• When RG port mastership changes from backup to master, access side BGP peers are brought up.
• It may take until the keepalive timer expires (for stateless ACLs) or the holdtime timer (for stateful ACLs) for
the new BGP peering to come up with the new RG port master.
When an NSG goes into controllerless mode, the VSC will re-advertise the access side BGP routes learned from this
NSG with local preference as -50 from the existing local preference (or default value of 100 if no local preference was
set). In this case, remote NSGs will prefer NSGs that are not in controllerless mode over those that are.
When an NSG in an RG goes into controllerless mode, the routes from backup NSG RG will be withdrawn or adver-
tised by the VSC with -50 local preference (depending on how the NSG goes into controllerless mode). This causes

6.1. BGP 327

 VNS User Guide, Release 5.3.2

the remote NSGs to switch to the master NSG RG as soon as the new master learns the BGP routes via its access side
peers.

Note: When two NSG are advertising the same route with a local preference set at a difference of 50, if the NSG with
the higher local preference goes controllerless mode the result is ECMP between the two remaining next hops.

MD5 Authentication

MD5 authentication is supported for LAN and WAN BGP peers. Authentication is performed between neighboring
routers before establishing the BGP session by verifying a password using the MD5 message-based digest. The
authentication key can be any combination of ASCII characters up to 255 characters long.
The authentication key can be configured in a BGP neighbor blob using the authentication-key parameter. See
Sample BGP Neighbor XML Blob (page 339) for more information.

Multi-hop

When enabling EBGP multi-hop support, you must configure static routes on the VSD to resolve the BGP peer IP
address. The NSG advertises these static routes by default to the BGP peers. In order to avoid peering issues, it is
recommended to add a BGP export policy to restrict the advertisement of these static routes or alternatively install
static routes with a prefix length that is shorter than or equal to the BGP peer subnet.
Multihop can be configured in a BGP neighbor blob using the multihop parameter. See Sample BGP Neighbor
XML Blob (page 339) for more information.

BGP Session Redundancy

The NSG can be configured with two BGP sessions to two different PEs on a single uplink. From 5.2.2, routes learnt
from BGP PE will be preferred over static routes pointing to Gateway (defined in Uplink connection). For example,
let us say that the gateway points to PE1 and the route to VSC is learnt from PE2. Traffic to VSC will be directed
towards PE2. The default route advertised by PE/s is handled as well. If the same route is received from both PEs, the
next-hop will be determined based on BGP attributes. The user can apply the import policy on the PE to prefer one
PE over the other. In the case of the active PE going down, routes from the other PE will be used. To avoid routes
received from PE1 being advertised to PE2, policies can be applied. For example, add an import policy on PE1 to tag
routes with community C1, import policy on PE2 to tag routes with community C2; export policy on PE1 to reject
routes with community C2 and export policy on PE2 to reject routes with community C1.
Dual PE redundancy is also supported on an uplink with secondary IP provisioned (post-bootstrap ONLY). Only a
single BGP peer can be associated to uplink connection for bootstrap. The second PE information will be downloaded
via Openflow/JSON). All traffic (except BGP session packets) will be sourced with Secondary IP. Please refer to
Secondary IP chapter for more details.
Until VNS 5.2.2, when PE redundancy was employed where an L2 underlay aggregated many CPEs and provided L2
underlay connectivity to two or more PEs, VRRP was used because it is transparent to the NSG. With release 5.2.2,
BGP too can be used. In other words, BGP route advertisements can provide redundancy for the NSG when it has
two BGP sessions to two separate PEs over a single physical uplink. The following graphic captioned “Single Uplink,
Dual PE” illustrates that use case, with textual descriptions relevant to the graphic below it.

6.1. BGP 328

 VNS User Guide, Release 5.3.2

Fig. 6.4: Single Uplink, Dual PE

Route policy
• PE1 “primary”
• PE2 “secondary”
• Option 1:
– Downstream traffic: NSG does AS path prepending towards PE2
– Upstream traffic: PE2 does AS path prepending towards NSG
• Option 2 (preferred):
– Downstream traffic: NSG sets MED 100 on primary BGP sessions advertised routes, MED 200 on sec-
ondary BGP session advertised routes
– Upstream traffic: NSG sets higher local preference on routes received from primary BGP session.

Note: When BGP is used on a redundant NSG vPort, it is recommended to create stateless ACL allow rules for
BGP, to improve the convergence time. The convergence time can be improved by having shorter keep-alive and hold
timers. For example, having keep-alive as 3 sec and hold-timer as 9 sec will have the peering to get established within
3-4sec if there is a mastership flap.

6.1. BGP 329

 VNS User Guide, Release 5.3.2

6.1.2 Default Route From LAN

From release 5.2.2, default route from BGP LAN is handled. Routes learnt from BGP LAN / VSD static routes / VSD
subnet routes are written in a new table called ‘uplink_rx’, which installs these routes with next-hop as svc-pat-tap
(OVS) .
Example

[root@nsg-103-131-142-142 ~]# ip route show table uplink_rx

default dev svc-pat-tap proto static scope link metric 1061243767

6.1.3 VIF RTM Routes

Remote NSG and NSG-UBR reachability is assessed per NSG separately. This assessment is made by the NSG in its
OVS data path computation, however, NSG BGP does not take NSG reachability into consideration while advertising
remote NSG routes to CE BGP peers. NSG BGP performs a separate NSG reachability check to determine whether a
route can be advertised downstream. This is done by maintaining a VIF RTM to resolve remote NSGs and NSG-UBRs.
The VIF keyword represents both NSGs and NSG-UBRs. The VIF RTM is used to determine underlay reachability
for NSG BGP route advertisement purposes. The VIF RTM populates its table with remote VIFs using information
from NAT and UBR routes. When resolving a BGP EVPN IP prefix, the next hop remote VIF is considered reachable
if a routing table lookup of the VIF RTM results in a match.
The VIF RTM table is updated as follows:
• When the NSG receives a NAT or UBR route, a VIF route is added to the VIF RTM if the route contains an
underlay common to one of the NSG uplinks.
• When the NSG-UBR receives a UBR route, a VIF route is added to the VIF RTM if the NSG-UBR is in a
common NSG-UBR group and belongs to one of the underlays of the source and destination NSGs.
Use the VIF RTM CLI Commands (page 377) to view VIF routes and assess route or underlay reachability.

6.1.4 BGP Configuration Workflow

Enable BGP and configure Local Autonomous System (AS)

Step 1 As CSPRoot, navigate to Organization Policies -> Organization Profiles, to create a new or modify
an existing profile.
Step 2 Check the Enable Routing Protocols option. Unless this option is enabled, BGP functionality
and related configuration options will not be available.

Note: Enterprise profiles that are in use cannot be edited.

Step 3 Create a new Organization (enterprise) and associate the BGP-enabled enterprise profile.

6.1. BGP 330

 VNS User Guide, Release 5.3.2

Fig. 6.5: Configure Local AS

Step 4 Configure Local AS for the enterprise, which is required when BGP is enabled and is used for all
neighbors created in this enterprise.

Note: For LAN and WAN BGP peer ASs and enterprise ASs, you can specify an AS number in
2-byte notation, or in 4-byte notation using the ASPLAIN format. The range for a 4-byte AS is 0 to
4,294,967,295.

6.1. BGP 331

 VNS User Guide, Release 5.3.2

Define Routing Policies

Routing policies can be created at Enterprise level or at the domain level.

• Any number of routing policies may be created.
• Only CSPRoot or Enterprise administrator may update/delete routing policies created at the enterprise level.
• A routing policy may be deleted only when it is not in use.

Note: The NSG floating default route (0.0.0.0/0 with pref 255) can be advertised or blocked depending on the applied
policy.

Routing Policy at Enterprise Level

Step 1 As Enterprise administrator, navigate to Infrastructure -> Routing Policies. Click on in

the panel to add a Routing Policy or edit an existing one.

6.1. BGP 332

 VNS User Guide, Release 5.3.2

Define Routing Policy

Step 2 In the popup, enter Name (upto 15 characters), Description, Default Route Action (Accept or
Reject), and a Policy Definition. Policy definition uses the blob (page 341) model. Default actions
in the blob take precedence over the one configured in the policy via the Default Route Action
dropdown. Refer to order of inheritance of default actions (page 345) for additional details.

Routing Policy at Domain Level

BGP-related information associated at the Domain instance level complements any definition by the Enterprise Admin
at the Domain Template or at the enterprise level.
The domain-specific routing policies will not be visible in other domain instances.

Step 1 As Network user, navigate to Networks -> L3 Domains, and select an instantiated Domain.

6.1. BGP 333

 VNS User Guide, Release 5.3.2

Step 2 From the far right panel, select icon, and click on to open new Routing Policy popup. Add
the new policy as described earlier.

Create BGP Profile

BGP routing profiles are used to assign default import / export policies and dampening configuration to the neighbors.
Profiles are assigned to Uplink Port VLAN for uplink/PE neighbors, and to domain templates for access side/CPE
neighbors. Only a CSPRoot or Enterprise administrator may delete a BGP profile and that only when it is not in use.

Step 1 As Enterprise administrator, navigate to Infrastructure -> BGP Profiles. Click on icon in
the panel to add a BGP Profile or edit an existing one.

6.1. BGP 334

 VNS User Guide, Release 5.3.2

Fig. 6.6: Create BGP Profile

Step 2 Enter Name (unique per enterprise) and Description.

Step 3 Associate Import and Export Routing Policy.
Step 4 Enter Dampening attributes. Dampening is supported to protect NSG CPU from unstable eBGP
peering sessions
• In the event of a route flap, the route penalty goes up by 1024.

6.1. BGP 335

 VNS User Guide, Release 5.3.2

• When the penalty goes above the Suppression Period, the routes are suppressed and are not
advertised or used. Default is 3000 mins.
• The Half-Life Period is the duration after which the penalty is reduced by half. Default is 15
mins.
• After the Maximum Suppress Period elapses, the suppressed route it reused and advertised.
Default is 60 mins.
• When the penalty is below the Route Reuse Limit, the route is used and advertised. Default is
750.

BGP Peering with PE/Uplink

• Peers can be configured in both uplinks.

• Multiple peers can be configured in a single uplink/VLAN
• Routing profiles can be applied to each uplink/VLAN. This gets applied to all the peers configured in that
particular uplink/VLAN.
• For traffic to PE, underlay flag must be enabled in the domains/subnets.

Step 1 As Enterprise administrator, navigate to Infrastructure -> Network Service Gateways -> NSG
instance -> Network Port -> VLAN.

BGP Configuration for Uplink

Step 2 Assign BGP Profile to Uplink: Select at the top of the right panel and click on to select a
BGP profile to assign.

Define Uplink BGP Neighbor

Step 1 Select at the top of the right panel.

Step 2 Click on to add a BGP neighbor.

6.1. BGP 336

 VNS User Guide, Release 5.3.2

Define Uplink Peer

Step 3 In the New BGP Neighbor popup, enter Name, Description, Peer IP, Peer AS.
Step 4 Additional neighbor parameters can be configured in the form of a Blob (page 339).
Step 5 Optionally specify Import and Export Routing Policy. If routing policy is referenced at BGP
Profile level as well as at BGP Neighbor level, then routing policy referenced at BGP Profile

6.1. BGP 337

 VNS User Guide, Release 5.3.2

will be evaluated first. In other words, import/export policies specified at the peer level in this
window are appended to the policies at the profile level. Refer to order of inheritance of default
actions (page 345) for additional details.

BGP Peering with CPE/Access

As Enterprise administrator, navigate to Networks -> L3 Domains, and select an L3 Domain Template.

Note: Routing profiles assigned to domain templates get applied to all the peers created at the subnet.

Note: Each NSG has a default 0.0.0.0/0 route with a preference of 255 in each RTM.

Assign BGP Profile to Domain Template

Step 1 Select at the top of the far right panel.

Step 2 Click on to select a BGP profile to assign.

Define Access-side BGP Neighbor

• Peer IP is mandatory for bridge vPort.

• Multiple peers can be configured in a single vPort.
Step 1 As Enterprise administrator, navigate to Networks -> Domain -> Zone -> Subnet.

Step 2 Select at the top of the far right panel. Click on the in the panel to add a BGP neighbor as
described earlier.

6.1.5 XML Configuration Requirements

• Prefix list names must not exceed 15 characters.

• Each match condition must have only one prefix list.
• When BGP peer is connected through the Resiliency port, then the keep-alive timer and hold timer of the BGP
peer should match the resiliency heartbeat timers. For example, if the heartbeat timer of resiliency is configured
to 1 second, then BGP peers keep-alive time should be 1 second and hold-timer should be 3 seconds.

6.1.6 CLI Show Command Sample

Refer to BGP CLI (page 353) chapter for all the relevant show commands and sample output.

6.1. BGP 338

 VNS User Guide, Release 5.3.2

6.1.7 BGP Blob

A blob is an opaque object (as seen by VSD) based on an extensible OpenConfig schema whereby configuration
elements are defined as part of an XML document. The user may define such elements in order to configure BGP
policies on the NSG ultimately controlling route-import/export behavior.

Sample BGP Neighbor XML Blob

<neighbor xmlns="alu:nuage:bgp:neighbor">
<config>
<description>Neighbor description goes here</description>
<advertise-inactive>false</advertise-inactive>
<disable-4byte-asn>false</disable-4byte-asn>
<disable-capability-negotiation>false</disable-capability-negotiation>
<local-preference>200</local-preference>
<next-hop-self>true</next-hop-self>
<remove-private-as>true</remove-private-as>
<set-med-out>1234</set-med-out>
<split-horizon>false</split-horizon>
</config>
<timers>
<connect-retry>1234</connect-retry>
<hold-time>1234</hold-time>
<keepalive>10</keepalive>
<minimum-advertisement-interval>20</minimum-advertisement-interval>
</timers>
<transport>
<passive-mode>false</passive-mode>
<authentication-key>key</authentication-key>
<multihop>5</multihop>
</transport>
</neighbor>

Attention: Passive mode must be set to false for access and uplink neighbors. NSG needs to initiate BGP session
with peer.

config — The config element and its parameters are optional. If undefined, true and false values listed in the
sample above are the default values for each parameter.
timer — The timer element and its parameters are optional.
connect-retry — Defines the length of time, in seconds, to wait for attempting to reconnect. The default is 60
seconds.
hold-time — Defines the length of time, in seconds, to maintain a TCP session if no keepalive messages are
received from the remote neighbor. The default is 60 seconds.
keepalive — Defines the interval, in seconds, to send session keepalive messages to the neighbor. The default is
10 seconds. A value of 0 disables keepalive messages.
minimum-advertisement-interval — Defines the minimum interval, in seconds, in which BGP policies are
reevaluated.
authentication-key — Defines an MD5 authentication key.
multihop — Defines the multi-hop TTL for EBGP multihop support.

6.1. BGP 339

 VNS User Guide, Release 5.3.2

BGP Neighbor XML tree

|-- neighbor
| |-- config
| | |-- description
| | |-- advertise-inactive
| | |-- disable-4byte-asn
| | |-- disable-capability-negotiation
| | |-- local-preference
| | |-- local-as
| | |-- next-hop-self
| | |-- remove-private-as
| | |-- set-med-out
| | |-- split-horizon
| | |-- prefix-limit
| | +-- host-advertisement
| |
| |-- timers
| | |-- connect-retry
| | |-- hold-time
| | |-- keepalive
| | +-- minimum-advertisement-interval
| |
| |-- transport
| | |-- mtu-discovery
| | |-- passive-mode
| | |-- authentication-key
| | |-- authentication-key-hash2
| | +-- multihop
| |
| +-- error-handling
| +-- treat-as-withdraw

XML Conversion (NSG)

group "20.0.0.100" ----→ Internally configured by NSG

local-as 200 -------→ Value configured for Local AS when Enterprise is created
neighbor 20.0.0.100
connect-retry 1234
keepalive 10
hold-time 1234
local-preference 200
min-route-advertisement 20
next-hop-self
local-as 200 -------→ Value configured for Local AS when Enterprise is created
peer-as 200 -------→ Value configured for Peer AS when BGP Neighbor is defined
advertise-inactive
path-mtu-discovery
split-horizon
exit
exit

6.1. BGP 340

 VNS User Guide, Release 5.3.2

Sample Routing Policy XML Blob

<?xml version="1.0"?>
<routing-policy xmlns="alu:nuage:bgp:routing:policy">
<defined-sets>
<prefix-sets>
<prefix-set>
<prefix-set-name>prefix_list_1</prefix-set-name>
<prefix>
<ip-prefix>1.1.1.1/32</ip-prefix>
<masklength-range>exact</masklength-range>
</prefix>
<prefix>
<ip-prefix>2.2.2.0/24</ip-prefix>
<masklength-range>24..28</masklength-range>
</prefix>
</prefix-set>
<prefix-set>
<prefix-set-name>prefix_list_2</prefix-set-name>
<prefix>
<ip-prefix>11.11.0.0/16</ip-prefix>
<masklength-range>16..24</masklength-range>
</prefix>
</prefix-set>
</prefix-sets>
<community-sets>
<community-set>
<community-set-name>comm_1</community-set-name>
<community>
<community-name>100:101</community-name>
</community>
<community>
<community-name>100:102</community-name>
</community>
<community>
<community-name>100:103 100:104</community-name>
</community>
</community-set>
<community-set>
<community-set-name>comm_2</community-set-name>
<community>
<community-name>101:101</community-name>
</community>
<community>
<community-name>101:102</community-name>
</community>
</community-set>
</community-sets>
<as-path-sets>
<as-path-set>
<as-path-set-name>as_path_1</as-path-set-name>
<as-path-expression>*100*</as-path-expression>
</as-path-set>
<as-path-set>
<as-path-set-name>as_path_2</as-path-set-name>
<as-path-expression>*200*</as-path-expression>
</as-path-set>
</as-path-sets>

6.1. BGP 341

 VNS User Guide, Release 5.3.2

</defined-sets>
<policy-definition>
<statements>
<statement>
<name>entry_1</name>
<conditions>
<match-prefix-set>
<prefix-set>prefix_list_1</prefix-set>
</match-prefix-set>
<match-community-set>
<community-set>comm_1</community-set>
</match-community-set>
<match-as-path-set>
<as-path-set>as_path_1</as-path-set>
</match-as-path-set>
<install-protocol-eq>BGP</install-protocol-eq>
</conditions>
<actions>
<accept-route-set>
<accept-route/>
<as-path-set>as_path_2</as-path-set>
<community-set>comm_2</community-set>
<local-preference>200</local-preference>
<metric>
<add>10</add>
<subtract>10</subtract>
<set>110</set>
</metric>
<next-hop>10.10.1.1</next-hop>
<next-hop-self>true</next-hop-self>
<preference>100</preference>
</accept-route-set>
<reject-route/>
</actions>
</statement>
<statement>
<name>entry_2</name>
<conditions>
<match-prefix-set>
<prefix-set>prefix_list_2</prefix-set>
</match-prefix-set>
<install-protocol-eq>STATIC</install-protocol-eq>
</conditions>
<actions>
<reject-route/>
</actions>
</statement>
</statements>
<default-action>
<accept-route-set>
<accept-route/>
<community-set>comm_1</community-set>
<local-preference>200</local-preference>
</accept-route-set>
<reject-route/>
</default-action>
</policy-definition>
</routing-policy>

6.1. BGP 342

 VNS User Guide, Release 5.3.2

Routing policy XML tree

|-- routing-policy
| |-- defined-sets
| | |-- prefix-sets
| | | +-- prefix-set
| | | |-- prefix-set-name
| | | +-- prefix
| | | |-- ip-prefix
| | | +-- masklength-range
| | |
| | |-- neighbor-sets
| | | +-- neighbor-set
| | | |-- neighbor-set-name
| | | +-- neighbor
| | | +-- address
| | |
| | |-- community-sets
| | | +-- community-set
| | | |-- community-set-name
| | | +-- community
| | | +-- community-name
| | |
| | +-- as-path-sets
| | +-- as-path-set
| | |-- as-path-set-name
| | +-- as-path-expression
| |
| +-- policy-definition
| |-- statements
| | +-- statement
| | |-- name
| | |-- conditions
| | | |-- match-prefix-set
| | | | +-- prefix-set
| | | |
| | | |-- match-neighbor-set
| | | | +-- neighbor-set
| | | |
| | | |-- match-community-set
| | | | +-- community-set
| | | |
| | | |-- match-as-path-set
| | | | +-- as-path-set
| | | |
| | | +-- install-protocol-eq
| | |
| | +-- actions
| | |-- accept-route-set
| | | |-- accept-route
| | | |-- as-path-set
| | | |-- community-set
| | | |-- community-remove
| | | |-- community-replace
| | | |-- local-preference
| | | |-- metric
| | | | |-- add
| | | | |-- subtract

6.1. BGP 343

 VNS User Guide, Release 5.3.2

| | | | +-- set
| | | |
| | | |-- next-hop
| | | |-- next-hop-self
| | | |-- preference
| | | +-- as-path-prepend
| | |
| | +-- reject-route
| |
| +-- default-action
| |-- accept-route-set
| | |-- accept-route
| | |-- as-path-set
| | |-- community-set
| | |-- community-remove
| | |-- community-replace
| | |-- local-preference
| | |-- metric
| | | |-- add
| | | |-- subtract
| | | +-- set
| | |
| | |-- next-hop
| | |-- next-hop-self
| | |-- preference
| | +-- as-path-prepend
| |
| +-- reject-route

XML Conversion (SROS)

prefix-list "RP-1_prefix_list_1"
prefix 1.1.1.1/32 exact
prefix 2.2.2.0/24 prefix-length-range 24-28
exit
prefix-list "RP-1_prefix_list_2"
prefix 11.11.0.0/16 prefix-length-range 16-24
exit
community "RP-1_comm_1" members "100:101" "100:102" "100:103"
as-path "RP-1_as_path_1" "100*"
policy-statement "RP-1"
entry 10
from
protocol static
prefix-list "RP-1_prefix_list_1"
as-path "RP-1_as_path_1"
community "RP-1_comm_1"
exit
action accept
as-path add "RP-1_as_path_2"
community add "RP-1_comm_2"
preference 100
metric set 110
next-hop-self
exit
exit

6.1. BGP 344

 VNS User Guide, Release 5.3.2

default-action accept
community add "RP-1_comm_1"
exit
exit

Order of Inheritance of Default Actions

BGP routing policies may be specified during different stages of the BGP workflow, such as in BGP Profile and/or
BGP Neighbor definition. Default action can be specified either by using the drop down menu in the routing policy
object or can be specified inside the policy definition blob.

Default action evaluation rules

Default action for routing policy evaluation follows an ordered inheritance model as follows:
• Rule 1: Default action specified in the policy definition blob will take precedence over the Default Route Action
in the policy object.
• Rule 2: If routing policy is assigned at BGP Profile level as well as at BGP Neighbor level, then routing policy
at BGP Profile level will be evaluated first.
If default action of routing policy at BGP Profile level (as evaluated per Rule 1) is:
• Accept, then routing policy at Neighbor level will be evaluated. If none of the entries of this policy match, then
default action of routing policy at neighbor level will be applied to the route (per Rule 2).
• Reject, then the route will be rejected. Routing policy at Neighbor level will not be evaluated in this case.
Similarly, if any entry of routing policy at BGP Profile level is matched and this entry’s action is accept, then routing
policy at Neighbor level will be evaluated. Else if action is reject, the route will be rejected and routing policy at
Neighbor level will not be evaluated.

6.1.8 BGP Policy Routing Use Cases

An NSG running a BGP instance learns routes from three sources:

• access-side BGP peers
• uplink BGP peers
• controller BGP peers forwarding routes from remote NSGs
Routing policies can be used to ensure that routes from a particular source are preferred over other routes. When
conflicting routes are received from different sources, routes learned from the access-side BGP peers are preferred by
default.
The following table lists the protocol preferences used by the controller and the NSG access-side RTM for the BGP
instance. These default values can be changes using routing policies by matching on specific routes. Note that a lower
preference value is preferred over a higher one.
VSC RTM NSG
Route Origin
Preference Protocol Preference Protocol
VSD-allocated subnet routes 0 NVC_LOCAL 0 NVC_LOCAL
VSD static routes 5 NVC_STATIC 5 NVC_LOCAL
Remote BGP-originated EVPN routes 170 BGP_VPN 170 BGP_VPN
LAN BGP routes from local NSG 170 BGP_VPN 160 BGP
WAN BGP routes from PE n/a n/a 170 BGP_VPN

6.1. BGP 345

 VNS User Guide, Release 5.3.2

On an NSG, the nuage-bgp-show rtm-routes command displays the following route types:
• Static: Default routes are programmed as static routes. Static routes are advertised to neighbors if
the protocol STATIC is allowed by routing policies.
• NVC_LOCAL: VSD-configured subnets and local /32 routes from those subnets.
• BGP: Routes learned from local BGPv4 peers.
• BGP_VPN: Remote routes in access-side domains learned from the controller; or, routes leaked
between uplink and access-side domains.
For an NSG BGP instance, an import routing policy can be used to the access-side or uplink BGP peer to
set a BGP protocol preference high or lower than the BGP VPN preference. A preference values applied
on the uplink VRF and the access-side domain VRF are retained when routes are leaked between the two.
For example, if an NSG should prefer a remote NSG route over a local BGP route, an import routing
policy can be applied on the BGP neighbor to increase the BGP protocol preference to a value greater
than 170 (the default BGP protocol preference for BGP VPN remote NSG routes).
If that NSG should prefer routes from an uplink neighbor over remote NSG routes, an import routing
policy can be applied on the uplink BGP peers to decrease the BGP protocol preference to a value lesser
than 170. If remote NSG routes should be preferred over uplink routes, the policy should increase the
BGP protocol preference to a value greater than 170.

Central Gateway

In this use case, the SD-WAN service is integrated with an isolated MPLS WAN and provides connectivity between
endpoints via a network of central gateways. In this case, the central gateways are redundant NSG-UBRs at a hub site.
By default, the offnets use ECMP to one of the NSG-UBRs to reach the legacy IPVPN networks.
If a specific NSG-UBR needs to be preferred, you can configure policies to ensure traffic is routed through that NSG-
UBR. There are multiple ways to configure the policy. If multiple BGP VPN routes exist with varied BGP attributes,

6.1. BGP 346

 VNS User Guide, Release 5.3.2

the controller does not collate these routes into ECMP routes, and the best route is selected.
The following use case examples show how offnet NSGs can route traffic to the NSG-UBR connected to the same
controller.

Example #1

Apply an import routing policy to the access-side neighbor behind the preferred UBR that tags the routes with a
community. Apply an import routing policy on the VSC that matches on the community and sets the preference to a
value lower than the default of 170 for these routes. Together, these policies route traffic to the preferred NSG-UBR
from NSG that share the same controller.
Note the community-set parameter in the following sample.
Sample BGP neighbor blob:

<?xml version="1.0" encoding="UTF-8"?>

<routing-policy xmlns="alu:nuage:bgp:routing:policy">
<defined-sets>
<prefix-sets>
<prefix-set>
<prefix-set-name>prefix_list_1</prefix-set-name>
<prefix>
<ip-prefix>1.1.1.1/32</ip-prefix>
<masklength-range>exact</masklength-range>
</prefix>
</prefix-set>
</prefix-sets>
<community-sets>
<community-set>
<community-set-name>community_list_1</community-set-name>
<community>
<community-name>1:1</community-name>
</community>
</community-set>
</community-sets>
</defined-sets>
<policy-definition>
<statements>
<statement>
<name>entry_1</name>
<conditions>
<match-prefix-set>
<prefix-set>prefix_list_1</prefix-set>
</match-prefix-set>
<install-protocol-eq>BGP</install-protocol-eq>
</conditions>
<actions>
<accept-route-set>
<accept-route />
<community-set>community_list_1</community-set>
</accept-route-set>
</actions>
</statement>
</statements>
<default-action>
<accept-route-set>
<accept-route />

6.1. BGP 347

 VNS User Guide, Release 5.3.2

</accept-route-set>
</default-action>
</policy-definition>
</routing-policy>

Sample VSC configuration:

*vsc>config>router>policy-options# info
----------------------------------------------
community "comm" members "1:1"
policy-statement "comm"
entry 10
from
community "comm"
exit
action accept
preference 160
exit
exit
default-action accept
exit
exit

*vsc>config>router>bgp# info
----------------------------------------------
family ipv4 vpn-ipv4 evpn
vpn-apply-import
connect-retry 5
hold-time 120
min-route-advertisement 1
import "comm"
router-id 10.20.1.7
rapid-withdrawal
rapid-update evpn
group "test"
neighbor 10.20.1.9
med-out 100
peer-as 1000
exit
exit
no shutdown

Example #2

Apply an import routing policy to the access-side neighbor behind the non-preferred UBR with a higher MED so that
other UBR learning routes with the default MED are preferred. This policy routes traffic to the preferred NSG-UBR
from all NSGs.
Note the metric parameter in the following sample.
Sample BGP neighbor blob:
<?xml version="1.0" encoding="UTF-8"?>
<routing-policy xmlns="alu:nuage:bgp:routing:policy">
<defined-sets>
<prefix-sets>
<prefix-set>

6.1. BGP 348

 VNS User Guide, Release 5.3.2

<prefix-set-name>prefix_list_1</prefix-set-name>
<prefix>
<ip-prefix>1.1.1.0/24</ip-prefix>
<masklength-range>exact</masklength-range>
</prefix>
</prefix-set>
</prefix-sets>
</defined-sets>
<policy-definition>
<statements>
<statement>
<name>entry_1</name>
<conditions>
<match-prefix-set>
<prefix-set>prefix_list_1</prefix-set>
</match-prefix-set>
<install-protocol-eq>BGP</install-protocol-eq>
</conditions>
<actions>
<accept-route-set>
<accept-route />
<metric>
<set>10</set>
</metric>
</accept-route-set>
</actions>
</statement>
</statements>
<default-action>
<accept-route-set>
<accept-route />
</accept-route-set>
</default-action>
</policy-definition>
</routing-policy>

Example #3

Apply an import routing policy to the access-side neighbor behind the non-preferred UBR with AS path prepend so
that other UBR learning routes with lower AS path length are preferred. This policy routes traffic to the preferred
NSG-UBR from all NSGs.
Note the as-path-set parameter in the following sample.
Sample BGP neighbor blob:
<?xml version="1.0" encoding="UTF-8"?>
<routing-policy xmlns="alu:nuage:bgp:routing:policy">
<defined-sets>
<prefix-sets>
<prefix-set>
<prefix-set-name>prefix_list_1</prefix-set-name>
<prefix>
<ip-prefix>1.1.1.0/24</ip-prefix>
<masklength-range>exact</masklength-range>
</prefix>
</prefix-set>

6.1. BGP 349

 VNS User Guide, Release 5.3.2

</prefix-sets>
<as-path-sets>
<as-path-set>
<as-path-set-name>as-path_list_1</as-path-set-name>
<as-path-expression>100</as-path-expression>
</as-path-set>
</as-path-sets>
</defined-sets>
<policy-definition>
<statements>
<statement>
<name>entry_1</name>
<conditions>
<match-prefix-set>
<prefix-set>prefix_list_1</prefix-set>
</match-prefix-set>
<install-protocol-eq>BGP</install-protocol-eq>
</conditions>
<actions>
<accept-route-set>
<accept-route />
<as-path-set>as-path_list_1</as-path-set>
</accept-route-set>
</actions>
</statement>
</statements>
<default-action>
<accept-route-set>
<accept-route />
</accept-route-set>
</default-action>
</policy-definition>
</routing-policy>

Example #4

If the central gateway forms an iBGP session with the IPVPN, this solution could be used.
Apply an import routing policy to the access-side neighbor behind the preferred UBR with a higher local preference.
This policy routes traffic to the preferred NSG-UBR from all NSGs.
Note the local-preference parameter in the following sample.
Sample BGP neighbor blob:

<?xml version="1.0" encoding="UTF-8"?>

<routing-policy xmlns="alu:nuage:bgp:routing:policy">
<defined-sets>
<prefix-sets>
<prefix-set>
<prefix-set-name>prefix_list_1</prefix-set-name>
<prefix>
<ip-prefix>1.1.1.0/24</ip-prefix>
<masklength-range>exact</masklength-range>
</prefix>
</prefix-set>
</prefix-sets>

6.1. BGP 350

 VNS User Guide, Release 5.3.2

</defined-sets>
<policy-definition>
<statements>
<statement>
<name>entry_1</name>
<conditions>
<match-prefix-set>
<prefix-set>prefix_list_1</prefix-set>
</match-prefix-set>
<install-protocol-eq>BGP</install-protocol-eq>
</conditions>
<actions>
<accept-route-set>
<accept-route />
<local-preference>150</local-preference>
</accept-route-set>
</actions>
</statement>
</statements>
<default-action>
<accept-route-set>
<accept-route />
</accept-route-set>
</default-action>
</policy-definition>
</routing-policy>

Distributed Interworking — LAN

In this use case, there are no fully isolated networks. Hybrid branch sites peer with the legacy IPVPN network on the
LAN side via BGP. By default, offnets use ECMP to both the NSG-UBR and the hybrid NSGs to reach the IPVPN.

6.1. BGP 351

 VNS User Guide, Release 5.3.2

To prefer routes through the NSG-UBR, a policy is required

Example 1: If the NSG-UBRs and hybrid NSGs form an iBGP session with the IPVPN, apply an import routing
policy to the access-side neighbor behind the preferred UBR with a higher local preference. This policy routes traffic
to the preferred NSG-UBR from all NSGs.
Example 2: Apply an import routing policy to the access-side neighbor behind the hybrid NSGs with a higher MED
so that other NSG-UBR learning routes with the default MED are preferred. This policy routes traffic to the preferred
NSG-UBR from all NSGs.
Consider the following in this use case:
• The hybrid site NSGs prefer local access side BGP routes for traffic to the legacy IPVPN networks.
• If the preferred NSG-UBR routes go down, the offnet site controllers automatically switch over to routes through
the hybrid site NSGs.

Distributed Interworking — WAN

This use case is similar to the previous one, except the hybrid site peers on the WAN side via BGP.
Route selection behavior in this use case is as follows:
Hybrid NSG traffic toward offnet NSG
• Overlay routes are received on the hybrid NSG via the following path: Offnet NSG - VSC - Hybrid NSG.
• Underlay routes are received on the hybrid NSG via the following path: Offnet NSG - VSC - NSG-UBR -
IPVPN - Hybrid NSG.
• Due to an AS path loop with underlay routes, only overlay routes are programmed into the hybrid NSG.
• To ensure that overlay routes are preferred over underlay routes, you must configure an export and import
routing policy. Configure an export routing policy that tags routes with a specific community and apply it to

6.1. BGP 352

 VNS User Guide, Release 5.3.2

the BGP peer of the NSG-UBR. Configure an import routing policy matching on the community to set an RTM
preference higher than the default 170 and apply it to the uplink BGP peer of the hybrid NSG.
Offnet NSG traffic toward hybrid NSG
• Overlay routes are received on the offnet NSG via the following path: Hybrid NSG - VSC - Offnet NSG.
• Overlay routes are also received on the offnet NSG via the following path: Hybrid NSG - IPVPN - NSG-UBR -
VSC - Offnet NSG.
• Due to an AS path loop with the second overlay route learning path, only routes from the first overlay route
learning path are programmed into the offnet NSG.
• To ensure that overlay routes learned from the hybrid NSG are preferred over routes learned from the NSG-
UBR, you must configure an export and import routing policy. Configure an export routing policy that tags
routes with a specific community and apply it to the uplink peer of the hybrid NSG. Configure an import routing
policy matching on the community to set a local preference lower than the default 100 and apply it to the access
side BGP peer of the NSG-UBR. When the hybrid NSG goes into controllerless mode, routes learned from the
NSG-UBR are used and the hybrid is still reachable via the underlay.
Hybrid NSG traffic toward IPVPN
• Underlay routes are received on the hybrid NSG via the following path: IPVPN - Hybrid NSG.
• Overlay routes are received on the hybrid NSG via the following path: IPVPN - NSG-UBR - VSC - Hybrid
NSG.
• To ensure that underlay routes are preferred over overlay routes received from the controller, you must configure
an import routing policy. Configure an import routing policy that sets an RTM preference lower than the default
170 and apply it to the uplink peer of the hybrid NSG.
IPVPN traffic toward hybrid NSG
• Routes are received on the IPVPN via the following path: Hybrid - IPVPN.
• Routes are also received on the IPVPN via the following path: Hybrid - VSC - NSG-UBR - IPVPN.
• Routes learned from both paths are installed, however underlays routes are preferred on the IPVPN due to a
lower AS path length. If routes from the first path are not available, routes from the second path are used.
• To ensure that routes from the hybrid NSG are preferred over routes from the NSG-UBR, you must configure
an export routing policy. Configure an export routing policy with a local preference that is higher than the
preference value for routes from the NSG-UBR and apply it to the uplink peer of the hybrid NSG.

6.1.9 BGP CLI Commands

*A:vsc# tools vswitch 10.15.1.254 command "nuage-bgp-show --h"

usage: nuage-bgp-show [-h] COMMAND [ARG [ARG ...]]

Nuage Bgp Ovsdb Show Commands

optional arguments:
-h, --help show this help message and exit

Commands Description Args:

---------------------------------------------------------------------
summary-all Show router bgp summary all (for all Vrfs) None
summary Show router bgp summary for specific Vrf DomainVrfId
summary-nei Show router bgp summary for specific Vrf DomainVrfId,
˓→NeighborIp

and neighbor

6.1. BGP 353

 VNS User Guide, Release 5.3.2

neighbor Show neighbor information DomainVrfId,

˓→NeighborIp
neighbor-detail Show neighbor information in detail DomainVrfId,
˓→NeighborIp

neighbor-recv Show bgp routes received from neighbor DomainVrfId,

˓→NeighborIp

neighbor-send Show bgp routes sent to neighbor DomainVrfId,

˓→NeighborIp

rtm-routes Show rtm routes DomainVrfId

route Show specific bgp route DomainVrfId, Prefix
routes Show bgp routes DomainVrfId
routes-detail Show bgp routes in detail DomainVrfId
bgp-config Show applied bgp config for a Vrf DomainVrfId
bgp-config-all Show applied bgp config for all Vrfs None
policy-config Show applied routing policy config None
vrf-map Show domain vrf to internal vrf mapping None
rib-in-show Show bgp rib in **expects internal VrfId** InternalVrfId
rib-out-show Show bgp rib out **expects internal VrfId** InternalVrfId
help Print help string None

COMMAND Command to use.

ARG Arguments to COMMAND.

*A:vsc# tools vswitch 10.15.1.254 command "nuage-bgp-show summary-all"

===============================================================================
BGP Summary
===============================================================================
Neighbor
ServiceId AS PktRcvd InQ Up/Down State|Rcv/Act/Sent (Addr Family)
PktSent OutQ
-------------------------------------------------------------------------------

20.0.0.3
Svc: 20001 300 1322 0 01h52m05s 1/0/8 (IPv4)
1502 0
20.0.0.100
Svc: 20001 200 1322 0 01h52m05s 1/0/9 (IPv4)
1504 0
-------------------------------------------------------------------------------

*A:vsc# tools vswitch 10.15.1.254 command "nuage-bgp-show summary 20001"

===============================================================================
BGP Router ID:78.129.167.75 AS:200 Local AS:200
===============================================================================
BGP Admin State : Up BGP Oper State : Up
Total Peer Groups : 2 Total Peers : 2
Total BGP Paths : 4 Total Path Memory : 552
Total IPv4 Remote Rts : 2 Total IPv4 Rem. Active Rts : 0
Total McIPv4 Remote Rts : 0 Total McIPv4 Rem. Active Rts: 0
Total IPv6 Remote Rts : 0 Total IPv6 Rem. Active Rts : 0
Total IPv4 Backup Rts : 0 Total IPv6 Backup Rts : 0

Total Supressed Rts : 0 Total Hist. Rts : 0

Total Decay Rts : 0

6.1. BGP 354

 VNS User Guide, Release 5.3.2

===============================================================================
BGP Summary
===============================================================================
Neighbor
AS PktRcvd InQ Up/Down State|Rcv/Act/Sent (Addr Family)
PktSent OutQ
-------------------------------------------------------------------------------
20.0.0.3
300 1351 0 01h54m32s 1/0/8 (IPv4)
1535 0
20.0.0.100
200 1351 0 01h54m32s 1/0/9 (IPv4)
1537 0
-------------------------------------------------------------------------------

*A:vsc# tools vswitch 10.15.1.254 command "nuage-bgp-show summary-nei 20001 20.0.0.3"

===============================================================================
BGP Router ID:78.129.167.75 AS:200 Local AS:200
===============================================================================
BGP Admin State : Up BGP Oper State : Up
Total Peer Groups : 2 Total Peers : 2
Total BGP Paths : 4 Total Path Memory : 552
Total IPv4 Remote Rts : 2 Total IPv4 Rem. Active Rts : 0
Total McIPv4 Remote Rts : 0 Total McIPv4 Rem. Active Rts: 0
Total IPv6 Remote Rts : 0 Total IPv6 Rem. Active Rts : 0
Total IPv4 Backup Rts : 0 Total IPv6 Backup Rts : 0

Total Supressed Rts : 0 Total Hist. Rts : 0

Total Decay Rts : 0

===============================================================================
BGP Summary
===============================================================================
Neighbor
AS PktRcvd InQ Up/Down State|Rcv/Act/Sent (Addr Family)
PktSent OutQ
-------------------------------------------------------------------------------
20.0.0.3
300 1361 0 01h55m24s 1/0/8 (IPv4)
1546 0
-------------------------------------------------------------------------------

*A:vsc# tools vswitch 10.15.1.254 command "nuage-bgp-show neighbor 20001 20.0.0.3"

===============================================================================
BGP Neighbor
===============================================================================
-------------------------------------------------------------------------------
Peer : 20.0.0.3
Group : 20.0.0.3
-------------------------------------------------------------------------------
Peer AS : 300 Peer Port : 57910
Peer Address : 20.0.0.3
Local AS : 200 Local Port : 179
Local Address : 20.0.0.1

6.1. BGP 355

 VNS User Guide, Release 5.3.2

Peer Type : External

State : Established Last State : Established
Last Event : recvKeepAlive
Last Error : Cease (Connection Collision Resolution)
Local Family : IPv4
Remote Family : IPv4
Hold Time : 90 Keep Alive : 30
Min Hold Time : 0
Active Hold Time : 15 Active Keep Alive : 5
Cluster Id : None
Preference : 170 Num of Update Flaps : 0
Recd. Paths : 1
IPv4 Recd. Prefixes : 1 IPv4 Active Prefixes : 0
IPv4 Suppressed Pfxs : 0 VPN-IPv4 Suppr. Pfxs : 0
VPN-IPv4 Recd. Pfxs : 0 VPN-IPv4 Active Pfxs : 0
Mc IPv4 Recd. Pfxs. : 0 Mc IPv4 Active Pfxs. : 0
Mc IPv4 Suppr. Pfxs : 0 IPv6 Suppressed Pfxs : 0
IPv6 Recd. Prefixes : 0 IPv6 Active Prefixes : 0
VPN-IPv6 Recd. Pfxs : 0 VPN-IPv6 Active Pfxs : 0
VPN-IPv6 Suppr. Pfxs : 0 L2-VPN Suppr. Pfxs : 0
L2-VPN Recd. Pfxs : 0 L2-VPN Active Pfxs : 0
MVPN-IPv4 Suppr. Pfxs: 0 MVPN-IPv4 Recd. Pfxs : 0
MVPN-IPv4 Active Pfxs: 0 MDT-SAFI Suppr. Pfxs : 0
MDT-SAFI Recd. Pfxs : 0 MDT-SAFI Active Pfxs : 0
FLOW-IPV4-SAFI Suppr*: 0 FLOW-IPV4-SAFI Recd.*: 0
FLOW-IPV4-SAFI Activ*: 0 Rte-Tgt Suppr. Pfxs : 0
Rte-Tgt Recd. Pfxs : 0 Rte-Tgt Active Pfxs : 0
Evpn Suppr. Pfxs : 0 Evpn Recd. Pfxs : 0
Evpn Active Pfxs : 0
Backup IPv4 Pfxs : 0 Backup IPv6 Pfxs : 0
Mc Vpn Ipv4 Recd. Pf*: 0 Mc Vpn Ipv4 Active P*: 0
Backup Vpn IPv4 Pfxs : 0 Backup Vpn IPv6 Pfxs : 0
Input Queue : 0 Output Queue : 0
i/p Messages : 1395 o/p Messages : 1585
i/p Octets : 26559 o/p Octets : 30225
i/p Updates : 1 o/p Updates : 2
TTL Security : Disabled Min TTL Value : n/a
Graceful Restart : Disabled Stale Routes Time : n/a
Advertise Inactive : Enabled Peer Tracking : Disabled
Advertise Label : None
Auth key chain : n/a
Disable Cap Nego : Disabled Bfd Enabled : Disabled
Flowspec Validate : Disabled Default Route Tgt : Disabled
Local Capability : RtRefresh MPBGP 4byte ASN
Remote Capability : RtRefresh MPBGP 4byte ASN
Local AddPath Capabi*: Disabled
Remote AddPath Capab*: Send - None
: Receive - None
Import Policy : None Specified / Inherited
Export Policy : _nuage_int_pol_exp_bgpvpn

-------------------------------------------------------------------------------
Neighbors : 1
===============================================================================
* indicates that the corresponding row element may have been truncated.

*A:vsc# tools vswitch 10.15.1.254 command "nuage-bgp-show neighbor-detail 20001 20.0.

˓→0.3"

6.1. BGP 356

 VNS User Guide, Release 5.3.2

===============================================================================
BGP Neighbor
===============================================================================
-------------------------------------------------------------------------------
Peer : 20.0.0.3
Group : 20.0.0.3
-------------------------------------------------------------------------------
Peer AS : 300 Peer Port : 57910
Peer Address : 20.0.0.3
Local AS : 200 Local Port : 179
Local Address : 20.0.0.1
Peer Type : External
State : Established Last State : Established
Last Event : recvKeepAlive
Last Error : Cease (Connection Collision Resolution)
Local Family : IPv4
Remote Family : IPv4
Connect Retry : 120 Local Pref. : 200
Min Route Advt. : 30 Min AS Orig. : 15
Multihop : 0 (Default) AS Override : Disabled
Damping : Disabled Loop Detect : Ignore
MED Out : No MED Out Authentication : None
Next Hop Self : Enabled AggregatorID Zero : Disabled
Remove Private : Disabled Passive : Enabled
Peer Identifier : 20.0.0.3 Fsm Est. Trans : 1
Fsm Est. Time : 02h00m01s InUpd Elap. Time : 03h46m13s
Prefix Limit : No Limit
Prefix Log Only : Disabled Prefix Threshold : 90
Hold Time : 90 Keep Alive : 30
Min Hold Time : 0
Active Hold Time : 15 Active Keep Alive : 5
Cluster Id : None Client Reflect : Disabled
Preference : 170 Num of Update Flaps : 0
Recd. Paths : 1
IPv4 Recd. Prefixes : 1 IPv4 Active Prefixes : 0
IPv4 Suppressed Pfxs : 0 VPN-IPv4 Suppr. Pfxs : 0
VPN-IPv4 Recd. Pfxs : 0 VPN-IPv4 Active Pfxs : 0
Mc IPv4 Recd. Pfxs. : 0 Mc IPv4 Active Pfxs. : 0
Mc IPv4 Suppr. Pfxs : 0 IPv6 Suppressed Pfxs : 0
IPv6 Recd. Prefixes : 0 IPv6 Active Prefixes : 0
VPN-IPv6 Recd. Pfxs : 0 VPN-IPv6 Active Pfxs : 0
VPN-IPv6 Suppr. Pfxs : 0 L2-VPN Suppr. Pfxs : 0
L2-VPN Recd. Pfxs : 0 L2-VPN Active Pfxs : 0
MVPN-IPv4 Suppr. Pfxs: 0 MVPN-IPv4 Recd. Pfxs : 0
MVPN-IPv4 Active Pfxs: 0 MDT-SAFI Suppr. Pfxs : 0
MDT-SAFI Recd. Pfxs : 0 MDT-SAFI Active Pfxs : 0
FLOW-IPV4-SAFI Suppr*: 0 FLOW-IPV4-SAFI Recd.*: 0
FLOW-IPV4-SAFI Activ*: 0 Rte-Tgt Suppr. Pfxs : 0
Rte-Tgt Recd. Pfxs : 0 Rte-Tgt Active Pfxs : 0
Evpn Suppr. Pfxs : 0 Evpn Recd. Pfxs : 0
Evpn Active Pfxs : 0
Backup IPv4 Pfxs : 0 Backup IPv6 Pfxs : 0
Mc Vpn Ipv4 Recd. Pf*: 0 Mc Vpn Ipv4 Active P*: 0
Backup Vpn IPv4 Pfxs : 0 Backup Vpn IPv6 Pfxs : 0
Input Queue : 0 Output Queue : 0
i/p Messages : 1416 o/p Messages : 1607
i/p Octets : 26958 o/p Octets : 30643

6.1. BGP 357

 VNS User Guide, Release 5.3.2

i/p Updates : 1 o/p Updates : 2

TTL Security : Disabled Min TTL Value : n/a
Graceful Restart : Disabled Stale Routes Time : n/a
Advertise Inactive : Enabled Peer Tracking : Disabled
Advertise Label : None
Auth key chain : n/a
Disable Cap Nego : Disabled Bfd Enabled : Disabled
Flowspec Validate : Disabled Default Route Tgt : Disabled
Local Capability : RtRefresh MPBGP 4byte ASN
Remote Capability : RtRefresh MPBGP 4byte ASN
Local AddPath Capabi*: Disabled
Remote AddPath Capab*: Send - None
: Receive - None
Import Policy : None Specified / Inherited
Export Policy : _nuage_int_pol_exp_bgpvpn

-------------------------------------------------------------------------------
Neighbors : 1
===============================================================================
* indicates that the corresponding row element may have been truncated.

*A:vsc# tools vswitch 10.15.1.254 command "nuage-bgp-show neighbor-recv 20001 20.0.0.3

˓→"

===============================================================================
BGP Router ID:78.129.167.75 AS:200 Local AS:200
===============================================================================
Legend -
Status codes : u - used, s - suppressed, h - history, d - decayed, * - valid
Origin codes : i - IGP, e - EGP, ? - incomplete, > - best, b - backup

===============================================================================
BGP IPv4 Routes
===============================================================================
Flag Network LocalPref MED
Nexthop Path-Id VPNLabel
As-Path
-------------------------------------------------------------------------------
*i 20.0.0.0/24 n/a None
20.0.0.3 None -
300
-------------------------------------------------------------------------------
Routes : 1
===============================================================================

*A:vsc# tools vswitch 10.15.1.254 command "nuage-bgp-show neighbor-send 20001 20.0.0.3

˓→"

===============================================================================
BGP Router ID:78.129.167.75 AS:200 Local AS:200
===============================================================================
Legend -
Status codes : u - used, s - suppressed, h - history, d - decayed, * - valid
Origin codes : i - IGP, e - EGP, ? - incomplete, > - best, b - backup

===============================================================================
BGP IPv4 Routes
===============================================================================
Flag Network LocalPref MED

6.1. BGP 358

 VNS User Guide, Release 5.3.2

Nexthop Path-Id VPNLabel

As-Path
-------------------------------------------------------------------------------
? 0.0.0.0/0 n/a 0
20.0.0.1 None -
200
? 20.0.1.0/24 n/a None
20.0.0.1 None -
200
? 20.0.2.0/24 n/a None
20.0.0.1 None -
200
? 20.0.3.0/24 n/a None
20.0.0.1 None -
200
? 20.0.4.0/24 n/a None
20.0.0.1 None -
200
? 20.0.5.0/24 n/a None
20.0.0.1 None -
200
? 20.0.6.0/24 n/a None
20.0.0.1 None -
200
? 20.0.7.0/24 n/a None
20.0.0.1 None -
200
-------------------------------------------------------------------------------
Routes : 8
===============================================================================

*A:vsc# tools vswitch 10.15.1.254 command "nuage-bgp-show rtm-routes 20001"

===============================================================================
Route Table (Router: Vrf 4)
===============================================================================
Dest Prefix[Flags] Type Proto Age Pref
Next Hop[Interface Name] Metric
-------------------------------------------------------------------------------
0.0.0.0/0 Remote Static 02h03m48s 0
Ukwn Interface 0
20.0.0.0/24 Local NVC 02h03m45s 0
0
20.0.1.0/24 Local NVC 02h03m45s 0
0
20.0.2.0/24 Local NVC 02h03m45s 0
0
20.0.3.0/24 Local NVC 02h03m45s 0
0
20.0.4.0/24 Local NVC 02h03m45s 0
0
20.0.5.0/24 Local NVC 02h03m45s 0
0
20.0.6.0/24 Local NVC 02h03m45s 0
0
20.0.7.0/24 Local NVC 02h03m45s 0

6.1. BGP 359

 VNS User Guide, Release 5.3.2

0
-------------------------------------------------------------------------------
No. of Routes: 9
Flags: L = LFA nexthop available B = BGP backup route available
n = Number of times nexthop is repeated * = Virtual Nexthop
===============================================================================

*A:vsc# tools vswitch 10.15.1.254 command "nuage-bgp-show route 20001 20.0.0.0/24"

===============================================================================
BGP Router ID:78.129.167.75 AS:200 Local AS:200
===============================================================================
Legend -
Status codes : u - used, s - suppressed, h - history, d - decayed, * - valid
Origin codes : i - IGP, e - EGP, ? - incomplete, > - best, b - backup

===============================================================================
BGP IPv4 Routes
===============================================================================
-------------------------------------------------------------------------------
Original Attributes

Network : 20.0.0.0/24
Nexthop : 20.0.0.3
Path Id : None
From : 20.0.0.3
Res. Nexthop : 20.0.0.3
Local Pref. : n/a Interface Name : NotAvailable
Aggregator AS : None Aggregator : None
Atomic Aggr. : Not Atomic MED : None
Community : No Community Members
Cluster : No Cluster Members
Originator Id : None Peer Router Id : 20.0.0.3
Fwd Class : None Priority : None
Flags : Valid IGP
Route Source : External
AS-Path : 300

Modified Attributes

Network : 20.0.0.0/24
Nexthop : 20.0.0.3
Path Id : None
From : 20.0.0.3
Res. Nexthop : 20.0.0.3
Local Pref. : None Interface Name : NotAvailable
Aggregator AS : None Aggregator : None
Atomic Aggr. : Not Atomic MED : None
Community : No Community Members
Cluster : No Cluster Members
Originator Id : None Peer Router Id : 20.0.0.3
Fwd Class : None Priority : None
Flags : Valid IGP
Route Source : External
AS-Path : 300

-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
Original Attributes

6.1. BGP 360

 VNS User Guide, Release 5.3.2

Network : 20.0.0.0/24
Nexthop : 20.0.0.100
Path Id : None
From : 20.0.0.100
Res. Nexthop : 20.0.0.100
Local Pref. : 100 Interface Name : NotAvailable
Aggregator AS : None Aggregator : None
Atomic Aggr. : Not Atomic MED : None
Community : No Community Members
Cluster : No Cluster Members
Originator Id : None Peer Router Id : 20.0.0.100
Fwd Class : None Priority : None
Flags : Valid IGP
Route Source : Internal
AS-Path : No As-Path

Modified Attributes

Network : 20.0.0.0/24
Nexthop : 20.0.0.100
Path Id : None
From : 20.0.0.100
Res. Nexthop : 20.0.0.100
Local Pref. : 100 Interface Name : NotAvailable
Aggregator AS : None Aggregator : None
Atomic Aggr. : Not Atomic MED : None
Community : No Community Members
Cluster : No Cluster Members
Originator Id : None Peer Router Id : 20.0.0.100
Fwd Class : None Priority : None
Flags : Valid IGP
Route Source : Internal
AS-Path : No As-Path

-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
Routes : 2
===============================================================================

*A:vsc# tools vswitch 10.15.1.254 command "nuage-bgp-show routes 20001"

===============================================================================
BGP Router ID:78.129.167.75 AS:200 Local AS:200
===============================================================================
Legend -
Status codes : u - used, s - suppressed, h - history, d - decayed, * - valid
Origin codes : i - IGP, e - EGP, ? - incomplete, > - best, b - backup

===============================================================================
BGP IPv4 Routes
===============================================================================
Flag Network LocalPref MED
Nexthop Path-Id VPNLabel
As-Path
-------------------------------------------------------------------------------
*i 20.0.0.0/24 None None
20.0.0.3 None -

6.1. BGP 361

 VNS User Guide, Release 5.3.2

300
*i 20.0.0.0/24 100 None
20.0.0.100 None -
No As-Path
-------------------------------------------------------------------------------
Routes : 2
===============================================================================
value = 0 = 0x0

*A:vsc# tools vswitch 10.15.1.254 command "nuage-bgp-show routes-detail 20001"

===============================================================================
BGP Router ID:78.129.167.75 AS:200 Local AS:200
===============================================================================
Legend -
Status codes : u - used, s - suppressed, h - history, d - decayed, * - valid
Origin codes : i - IGP, e - EGP, ? - incomplete, > - best, b - backup

===============================================================================
BGP IPv4 Routes
===============================================================================
-------------------------------------------------------------------------------
Original Attributes

Network : 20.0.0.0/24
Nexthop : 20.0.0.3
Path Id : None
From : 20.0.0.3
Res. Nexthop : 20.0.0.3
Local Pref. : n/a Interface Name : NotAvailable
Aggregator AS : None Aggregator : None
Atomic Aggr. : Not Atomic MED : None
Community : No Community Members
Cluster : No Cluster Members
Originator Id : None Peer Router Id : 20.0.0.3
Fwd Class : None Priority : None
Flags : Valid IGP
Route Source : External
AS-Path : 300

Modified Attributes

Network : 20.0.0.0/24
Nexthop : 20.0.0.3
Path Id : None
From : 20.0.0.3
Res. Nexthop : 20.0.0.3
Local Pref. : None Interface Name : NotAvailable
Aggregator AS : None Aggregator : None
Atomic Aggr. : Not Atomic MED : None
Community : No Community Members
Cluster : No Cluster Members
Originator Id : None Peer Router Id : 20.0.0.3
Fwd Class : None Priority : None
Flags : Valid IGP
Route Source : External
AS-Path : 300

-------------------------------------------------------------------------------

6.1. BGP 362

 VNS User Guide, Release 5.3.2

-------------------------------------------------------------------------------
Original Attributes

Network : 20.0.0.0/24
Nexthop : 20.0.0.100
Path Id : None
From : 20.0.0.100
Res. Nexthop : 20.0.0.100
Local Pref. : 100 Interface Name : NotAvailable
Aggregator AS : None Aggregator : None
Atomic Aggr. : Not Atomic MED : None
Community : No Community Members
Cluster : No Cluster Members
Originator Id : None Peer Router Id : 20.0.0.100
Fwd Class : None Priority : None
Flags : Valid IGP
Route Source : Internal
AS-Path : No As-Path

Modified Attributes

Network : 20.0.0.0/24
Nexthop : 20.0.0.100
Path Id : None
From : 20.0.0.100
Res. Nexthop : 20.0.0.100
Local Pref. : 100 Interface Name : NotAvailable
Aggregator AS : None Aggregator : None
Atomic Aggr. : Not Atomic MED : None
Community : No Community Members
Cluster : No Cluster Members
Originator Id : None Peer Router Id : 20.0.0.100
Fwd Class : None Priority : None
Flags : Valid IGP
Route Source : Internal
AS-Path : No As-Path

-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
Routes : 2
===============================================================================

*A:vsc# tools vswitch 10.15.1.254 command "nuage-bgp-show bgp-config 20001"

autonomous-system 200
router-id 78.129.167.75
route-distinguisher 65534:12148

bgp
export "_nuage_int_pol_exp_bgpvpn"
group "20.0.0.3"
local-as 200
neighbor 20.0.0.3
connect-retry 120
keepalive 30
hold-time 90
local-preference 200
min-route-advertisement 30

6.1. BGP 363

 VNS User Guide, Release 5.3.2

next-hop-self
passive
local-as 200
peer-as 300
advertise-inactive
path-mtu-discovery
split-horizon
exit
exit
group "20.0.0.100"
local-as 200
neighbor 20.0.0.100
connect-retry 120
keepalive 30
hold-time 90
local-preference 200
min-route-advertisement 30
next-hop-self
passive
local-as 200
peer-as 200
advertise-inactive
path-mtu-discovery
split-horizon
exit
exit
no shutdown
exit
value = 0 = 0x0

*A:vsc# tools vswitch 10.15.1.254 command "nuage-bgp-show bgp-config-all"

===========================================
BGP Config Vrf: 1 RTM Vrf: 1
===========================================
autonomous-system 200
router-id 78.129.167.75
bgp
no shutdown
exit

===========================================
BGP Config Vrf: 2 RTM Vrf: 5
===========================================

===========================================
BGP Config Vrf: 3 RTM Vrf: 3
===========================================

===========================================
BGP Config Vrf: 20001 RTM Vrf: 4
===========================================
autonomous-system 200
router-id 78.129.167.75
route-distinguisher 65534:12148

bgp
export "_nuage_int_pol_exp_bgpvpn"
group "20.0.0.3"

6.1. BGP 364

 VNS User Guide, Release 5.3.2

local-as 200
neighbor 20.0.0.3
connect-retry 120
keepalive 30
hold-time 90
local-preference 200
min-route-advertisement 30
next-hop-self
passive
local-as 200
peer-as 300
advertise-inactive
path-mtu-discovery
split-horizon
exit
exit
group "20.0.0.100"
local-as 200
neighbor 20.0.0.100
connect-retry 120
keepalive 30
hold-time 90
local-preference 200
min-route-advertisement 30
next-hop-self
passive
local-as 200
peer-as 200
advertise-inactive
path-mtu-discovery
split-horizon
exit
exit
no shutdown
exit

===========================================
BGP Config Vrf: 20013 RTM Vrf: 2
===========================================

*A:vsc# tools vswitch 10.15.1.254 command "nuage-bgp-show policy-config"

=================================
Router Policy Config
=================================
policy-statement "_nuage_int_pol_exp_bgpvpn"
entry 10
from
protocol bgp-vpn
exit
action accept
exit
exit
entry 20
from
protocol nvc-local
exit

6.1. BGP 365

 VNS User Guide, Release 5.3.2

action accept
exit
exit
entry 30
from
protocol nvc-static
exit
action accept
exit
exit
entry 40
from
protocol nvc
exit
action accept
exit
exit
default-action accept
exit
exit
=================================

value = 35 = 0x23 = '#'

*A:vsc# tools vswitch 10.15.1.254 command "nuage-bgp-show vrf-map"

Domain Vrf to Rtm Vrf conversion map: Count: 4
Domain Vrf Rtm Vrf
2 5
3 3
20001 4
20013 2

*A:vsc# tools vswitch 10.15.1.254 command "nuage-bgp-show rib-in-show 4"

IPV4 RIB-IN:
Route: 0.0.0.0/0
my ptr 0xf514f550: route in head ptr 0xf0987f18: In Rtm/Ttm: NO/NO: Backup: NO:
˓→ Seq# 2

Info (0xf0987f18): Local-Peer-RTM: addr type IPV4: label 16777215: pref 0: owner
˓→STATIC: flags IMPORTED USED BEST : next (nil): attr 0xf2203228, orig 0xf2203228,

˓→path- id: 0

Attr: my ptr 0xf2203228: IPV4: ref 2: flags MED_PRSNT FC_PRSNT IMPORTED : num asn 0:
˓→as-len: 0:

len 136: asp len 0: comm len 0

ext_comm len 0: cluster len 0; pmsi len 0; tot_uta len 0
CRC checksum = 0xd2fc5987: asp ptr 0xf22032b0: comm ptr 0xf22032b0
ext_comm ptr 0xf22032b0: cluster_ptr 0xf22032b0: pmsi_ptr 0xf22032b0: uta
˓→0xf22032b0

nexthop 0.0.0.0
prio_modified 0: fc_modified 0: fc 0: priority:0
med 0: loc pref 0: aggr ip 0.0.0.0: aggr as 0: origin 2
AIGP 0 0, 0 0
Orig ID 0.0.0.0: rt_pref 0
igp_type 0: igp-inst: 0: opaque-igp-data: 0x0: proto-data: 0x0
orig_ifid 0x0: vr_id 0: orig_rcvd_lbl: 0
global_if_id 0x0:

6.1. BGP 366

 VNS User Guide, Release 5.3.2

Route: 20.0.0.0/24
my ptr 0xf514f520: route in head ptr 0xf0987118: In Rtm/Ttm: NO/NO: Backup: NO:
˓→ Seq# 47

Info (0xf0987118): Local-Peer-RTM: addr type IPV4: label 16777215: pref 0: owner NVC_
˓→LOCAL: flags IMPORTED USED BEST : next 0xf09871c0: attr 0xf2202e38, orig

˓→0xf2202e38, path-id: 0

Attr: my ptr 0xf2202e38: IPV4: ref 16: flags IMPORTED : num asn 0: as-len: 0:
len 136: asp len 0: comm len 0
ext_comm len 0: cluster len 0; pmsi len 0; tot_uta len 0
CRC checksum = 0xd2fc5987: asp ptr 0xf2202ec0: comm ptr 0xf2202ec0
ext_comm ptr 0xf2202ec0: cluster_ptr 0xf2202ec0: pmsi_ptr 0xf2202ec0: uta
˓→0xf2202ec0

nexthop 0.0.0.0
prio_modified 0: fc_modified 0: fc 0: priority:0
med 0: loc pref 0: aggr ip 0.0.0.0: aggr as 0: origin 2
AIGP 0 0, 0 0
Orig ID 0.0.0.0: rt_pref 0
igp_type 0: igp-inst: 0: opaque-igp-data: 0x0: proto-data: 0x0
orig_ifid 0x0: vr_id 0: orig_rcvd_lbl: 0
global_if_id 0x0:
Info (0xf09871c0): VR 4: Group 20.0.0.3: Peer 20.0.0.3: addr type IPV4: label
˓→16777215: pref 170: owner BGP: flags NONE ADV-INACT : next 0xf0987188: attr

˓→0xf2209270, orig 0xf2209270, path-id: 0

Attr: my ptr 0xf2209270: IPV4: ref 2: flags EBGP_LEARNED : num asn 1: as-len: 1:
nh_p 0xf3571d18: nh flags IPV4 RESOLVED
res nexthop 20.0.0.3: num_nh 1: metrics 0: out_ifid 0x0
len 144: asp len 8: comm len 0
ext_comm len 0: cluster len 0; pmsi len 0; tot_uta len 0
CRC checksum = 0x14767675: asp ptr 0xf22092f8: comm ptr 0xf2209300
ext_comm ptr 0xf2209300: cluster_ptr 0xf2209300: pmsi_ptr 0xf2209300: uta
˓→0xf2209300

nexthop 20.0.0.3
prio_modified 0: fc_modified 0: fc 0: priority:0
med 0: loc pref 0: aggr ip 0.0.0.0: aggr as 0: origin 0
AIGP 0 0, 0 0
Orig ID 0.0.0.0: rt_pref 0
igp_type 0: igp-inst: 0: opaque-igp-data: 0x0: proto-data: 0x0
orig_ifid 0x0: vr_id 0: orig_rcvd_lbl: 0
global_if_id 0x0:
AS_SEQUENCE: 1: AS #s 300
Info (0xf0987188): VR 4: Group 20.0.0.100: Peer 20.0.0.100: addr type IPV4: label
˓→16777215: pref 170: owner BGP: flags NONE : next (nil): attr 0xf2202c88, orig

˓→0xf2202c88, path-id: 0

Attr: my ptr 0xf2202c88: IPV4: ref 2: flags LOC_PREF_PRSNT : num asn 0: as-len: 0:
nh_p 0xf3573800: nh flags IPV4 RESOLVED
res nexthop 20.0.0.100: num_nh 1: metrics 0: out_ifid 0x0
len 136: asp len 0: comm len 0
ext_comm len 0: cluster len 0; pmsi len 0; tot_uta len 0
CRC checksum = 0x43b3e111: asp ptr 0xf2202d10: comm ptr 0xf2202d10
ext_comm ptr 0xf2202d10: cluster_ptr 0xf2202d10: pmsi_ptr 0xf2202d10: uta
˓→0xf2202d10

nexthop 20.0.0.100
prio_modified 0: fc_modified 0: fc 0: priority:0
med 0: loc pref 100: aggr ip 0.0.0.0: aggr as 0: origin 0

6.1. BGP 367

 VNS User Guide, Release 5.3.2

AIGP 0 0, 0 0
Orig ID 0.0.0.0: rt_pref 0
igp_type 0: igp-inst: 0: opaque-igp-data: 0x0: proto-data: 0x0
orig_ifid 0x0: vr_id 0: orig_rcvd_lbl: 0
global_if_id 0x0:

Route: 20.0.1.0/24
my ptr 0xf514ed70: route in head ptr 0xf0986350: In Rtm/Ttm: NO/NO: Backup: NO:
˓→ Seq# 39

Info (0xf0986350): Local-Peer-RTM: addr type IPV4: label 16777215: pref 0: owner NVC_
˓→LOCAL: flags IMPORTED USED BEST : next (nil): attr 0xf2202e38, orig 0xf2202e38,

˓→path-id: 0

Attr: my ptr 0xf2202e38: IPV4: ref 16: flags IMPORTED : num asn 0: as-len: 0:
len 136: asp len 0: comm len 0
ext_comm len 0: cluster len 0; pmsi len 0; tot_uta len 0
CRC checksum = 0xd2fc5987: asp ptr 0xf2202ec0: comm ptr 0xf2202ec0
ext_comm ptr 0xf2202ec0: cluster_ptr 0xf2202ec0: pmsi_ptr 0xf2202ec0: uta
˓→0xf2202ec0

nexthop 0.0.0.0
prio_modified 0: fc_modified 0: fc 0: priority:0
med 0: loc pref 0: aggr ip 0.0.0.0: aggr as 0: origin 2
AIGP 0 0, 0 0
Orig ID 0.0.0.0: rt_pref 0
igp_type 0: igp-inst: 0: opaque-igp-data: 0x0: proto-data: 0x0
orig_ifid 0x0: vr_id 0: orig_rcvd_lbl: 0
global_if_id 0x0:

Route: 20.0.2.0/24
my ptr 0xf514ec20: route in head ptr 0xf0987c78: In Rtm/Ttm: NO/NO: Backup: NO:
˓→ Seq# 40

Info (0xf0987c78): Local-Peer-RTM: addr type IPV4: label 16777215: pref 0: owner NVC_
˓→LOCAL: flags IMPORTED USED BEST : next (nil): attr 0xf2202e38, orig 0xf2202e38,

˓→path-id: 0

Attr: my ptr 0xf2202e38: IPV4: ref 16: flags IMPORTED : num asn 0: as-len: 0:
len 136: asp len 0: comm len 0
ext_comm len 0: cluster len 0; pmsi len 0; tot_uta len 0
CRC checksum = 0xd2fc5987: asp ptr 0xf2202ec0: comm ptr 0xf2202ec0
ext_comm ptr 0xf2202ec0: cluster_ptr 0xf2202ec0: pmsi_ptr 0xf2202ec0: uta
˓→0xf2202ec0

nexthop 0.0.0.0
prio_modified 0: fc_modified 0: fc 0: priority:0
med 0: loc pref 0: aggr ip 0.0.0.0: aggr as 0: origin 2
AIGP 0 0, 0 0
Orig ID 0.0.0.0: rt_pref 0
igp_type 0: igp-inst: 0: opaque-igp-data: 0x0: proto-data: 0x0
orig_ifid 0x0: vr_id 0: orig_rcvd_lbl: 0
global_if_id 0x0:

Route: 20.0.3.0/24
my ptr 0xf514ece0: route in head ptr 0xf09862e0: In Rtm/Ttm: NO/NO: Backup: NO:
˓→ Seq# 41

Info (0xf09862e0): Local-Peer-RTM: addr type IPV4: label 16777215: pref 0: owner NVC:
˓→flags IMPORTED USED BEST : next (nil): attr 0xf2202e38, orig 0xf2202e38, path- id:
˓→0

Attr: my ptr 0xf2202e38: IPV4: ref 16: flags IMPORTED : num asn 0: as-len: 0:

6.1. BGP 368

 VNS User Guide, Release 5.3.2

len 136: asp len 0: comm len 0

ext_comm len 0: cluster len 0; pmsi len 0; tot_uta len 0
CRC checksum = 0xd2fc5987: asp ptr 0xf2202ec0: comm ptr 0xf2202ec0
ext_comm ptr 0xf2202ec0: cluster_ptr 0xf2202ec0: pmsi_ptr 0xf2202ec0: uta
˓→0xf2202ec0

nexthop 0.0.0.0
prio_modified 0: fc_modified 0: fc 0: priority:0
med 0: loc pref 0: aggr ip 0.0.0.0: aggr as 0: origin 2
AIGP 0 0, 0 0
Orig ID 0.0.0.0: rt_pref 0
igp_type 0: igp-inst: 0: opaque-igp-data: 0x0: proto-data: 0x0
orig_ifid 0x0: vr_id 0: orig_rcvd_lbl: 0
global_if_id 0x0:

Route: 20.0.4.0/24
my ptr 0xf514f3a0: route in head ptr 0xf09862a8: In Rtm/Ttm: NO/NO: Backup: NO:
˓→ Seq# 42

Info (0xf09862a8): Local-Peer-RTM: addr type IPV4: label 16777215: pref 0: owner NVC:
˓→flags IMPORTED USED BEST : next (nil): attr 0xf2202e38, orig 0xf2202e38, path- id:
˓→0

Attr: my ptr 0xf2202e38: IPV4: ref 16: flags IMPORTED : num asn 0: as-len: 0:
len 136: asp len 0: comm len 0
ext_comm len 0: cluster len 0; pmsi len 0; tot_uta len 0
CRC checksum = 0xd2fc5987: asp ptr 0xf2202ec0: comm ptr 0xf2202ec0
ext_comm ptr 0xf2202ec0: cluster_ptr 0xf2202ec0: pmsi_ptr 0xf2202ec0: uta
˓→0xf2202ec0

nexthop 0.0.0.0
prio_modified 0: fc_modified 0: fc 0: priority:0
med 0: loc pref 0: aggr ip 0.0.0.0: aggr as 0: origin 2
AIGP 0 0, 0 0
Orig ID 0.0.0.0: rt_pref 0
igp_type 0: igp-inst: 0: opaque-igp-data: 0x0: proto-data: 0x0
orig_ifid 0x0: vr_id 0: orig_rcvd_lbl: 0
global_if_id 0x0:

Route: 20.0.5.0/24
my ptr 0xf514ed10: route in head ptr 0xf0986cb8: In Rtm/Ttm: NO/NO: Backup: NO:
˓→ Seq# 43

Info (0xf0986cb8): Local-Peer-RTM: addr type IPV4: label 16777215: pref 0: owner NVC_
˓→LOCAL: flags IMPORTED USED BEST : next (nil): attr 0xf2202e38, orig 0xf2202e38,

˓→path-id: 0

Attr: my ptr 0xf2202e38: IPV4: ref 16: flags IMPORTED : num asn 0: as-len: 0:
len 136: asp len 0: comm len 0
ext_comm len 0: cluster len 0; pmsi len 0; tot_uta len 0
CRC checksum = 0xd2fc5987: asp ptr 0xf2202ec0: comm ptr 0xf2202ec0
ext_comm ptr 0xf2202ec0: cluster_ptr 0xf2202ec0: pmsi_ptr 0xf2202ec0: uta
˓→0xf2202ec0

nexthop 0.0.0.0
prio_modified 0: fc_modified 0: fc 0: priority:0
med 0: loc pref 0: aggr ip 0.0.0.0: aggr as 0: origin 2
AIGP 0 0, 0 0
Orig ID 0.0.0.0: rt_pref 0
igp_type 0: igp-inst: 0: opaque-igp-data: 0x0: proto-data: 0x0
orig_ifid 0x0: vr_id 0: orig_rcvd_lbl: 0
global_if_id 0x0:

6.1. BGP 369

 VNS User Guide, Release 5.3.2

Route: 20.0.6.0/24
my ptr 0xf514eb60: route in head ptr 0xf0987d20: In Rtm/Ttm: NO/NO: Backup: NO:
˓→ Seq# 44

Info (0xf0987d20): Local-Peer-RTM: addr type IPV4: label 16777215: pref 0: owner NVC_
˓→LOCAL: flags IMPORTED USED BEST : next (nil): attr 0xf2202e38, orig 0xf2202e38,

˓→path-id: 0

Attr: my ptr 0xf2202e38: IPV4: ref 16: flags IMPORTED : num asn 0: as-len: 0:
len 136: asp len 0: comm len 0
ext_comm len 0: cluster len 0; pmsi len 0; tot_uta len 0
CRC checksum = 0xd2fc5987: asp ptr 0xf2202ec0: comm ptr 0xf2202ec0
ext_comm ptr 0xf2202ec0: cluster_ptr 0xf2202ec0: pmsi_ptr 0xf2202ec0: uta
˓→0xf2202ec0

nexthop 0.0.0.0
prio_modified 0: fc_modified 0: fc 0: priority:0
med 0: loc pref 0: aggr ip 0.0.0.0: aggr as 0: origin 2
AIGP 0 0, 0 0
Orig ID 0.0.0.0: rt_pref 0
igp_type 0: igp-inst: 0: opaque-igp-data: 0x0: proto-data: 0x0
orig_ifid 0x0: vr_id 0: orig_rcvd_lbl: 0
global_if_id 0x0:

Route: 20.0.7.0/24
my ptr 0xf514f7c0: route in head ptr 0xf0987428: In Rtm/Ttm: NO/NO: Backup: NO:
˓→ Seq# 45

Info (0xf0987428): Local-Peer-RTM: addr type IPV4: label 16777215: pref 0: owner NVC:
˓→flags IMPORTED USED BEST : next (nil): attr 0xf2202e38, orig 0xf2202e38, path- id:
˓→0

Attr: my ptr 0xf2202e38: IPV4: ref 16: flags IMPORTED : num asn 0: as-len: 0:
len 136: asp len 0: comm len 0
ext_comm len 0: cluster len 0; pmsi len 0; tot_uta len 0
CRC checksum = 0xd2fc5987: asp ptr 0xf2202ec0: comm ptr 0xf2202ec0
ext_comm ptr 0xf2202ec0: cluster_ptr 0xf2202ec0: pmsi_ptr 0xf2202ec0: uta
˓→0xf2202ec0

nexthop 0.0.0.0
prio_modified 0: fc_modified 0: fc 0: priority:0
med 0: loc pref 0: aggr ip 0.0.0.0: aggr as 0: origin 2
AIGP 0 0, 0 0
Orig ID 0.0.0.0: rt_pref 0
igp_type 0: igp-inst: 0: opaque-igp-data: 0x0: proto-data: 0x0
orig_ifid 0x0: vr_id 0: orig_rcvd_lbl: 0
global_if_id 0x0:

value = 10 = 0xa

*A:vsc# tools vswitch 10.15.1.254 command "nuage-bgp-show rib-out-show 4"

IPV4 RIB-OUT:
Route: 0.0.0.0/0
my ptr 0xf514f550: ro_head ptr 0xee1bbd50 label: 16777215
Info (0xee1bbd50): Tribe 0xf5165928: Bitmap 0x1: Mod-Bitmap 0x0: next 0xee1bbc00:
˓→label: 16777215 attr 0xf3e57870: withdraw 0

rcvd-path-id: 0 (0x0) advt-path-id: 0

Sent to:
VR 4: Group 20.0.0.3: Peer 20.0.0.3(0xf51ab7d0):

6.1. BGP 370

 VNS User Guide, Release 5.3.2

Attr: my ptr 0xf3e57870: IPV4: ref 1: flags MED_PRSNT FC_PRSNT : num asn 1: as-len: 1:
len 144: asp len 8: comm len 0
ext_comm len 0: cluster len 0; pmsi len 0; tot_uta len 0
CRC checksum = 0xb91081ab: asp ptr 0xf3e578f8: comm ptr 0xf3e57900
ext_comm ptr 0xf3e57900: cluster_ptr 0xf3e57900: pmsi_ptr 0xf3e57900: uta
˓→0xf3e57900

nexthop 0.0.0.0
prio_modified 0: fc_modified 0: fc 0: priority:0
med 0: loc pref 0: aggr ip 0.0.0.0: aggr as 0: origin 2
AIGP 0 0, 0 0
Orig ID 0.0.0.0: rt_pref 0
igp_type 0: igp-inst: 0: opaque-igp-data: 0x0: proto-data: 0x0
orig_ifid 0x0: vr_id 0: orig_rcvd_lbl: 0
global_if_id 0x0:
AS_SEQUENCE: 1: AS #s 200

Info (0xee1bbc00): Tribe 0xf5164548: Bitmap 0x1: Mod-Bitmap 0x0: next (nil): label:
˓→16777215 attr 0xf51ae968: withdraw 0

rcvd-path-id: 0 (0x0) advt-path-id: 0

Sent to:
VR 4: Group 20.0.0.100: Peer 20.0.0.100(0xf51a9690):

Attr: my ptr 0xf51ae968: IPV4: ref 1: flags MED_PRSNT FC_PRSNT LOC_PREF_PRSNT : num
˓→asn 0: as-len: 0:

len 136: asp len 0: comm len 0

ext_comm len 0: cluster len 0; pmsi len 0; tot_uta len 0
CRC checksum = 0xf9634e2e: asp ptr 0xf51ae9f0: comm ptr 0xf51ae9f0
ext_comm ptr 0xf51ae9f0: cluster_ptr 0xf51ae9f0: pmsi_ptr 0xf51ae9f0: uta
˓→0xf51ae9f0

nexthop 0.0.0.0
prio_modified 0: fc_modified 0: fc 0: priority:0
med 0: loc pref 200: aggr ip 0.0.0.0: aggr as 0: origin 2
AIGP 0 0, 0 0
Orig ID 0.0.0.0: rt_pref 0
igp_type 0: igp-inst: 0: opaque-igp-data: 0x0: proto-data: 0x0
orig_ifid 0x0: vr_id 0: orig_rcvd_lbl: 0
global_if_id 0x0:

Route: 20.0.0.0/24
my ptr 0xf514f520: ro_head ptr 0xee1bb6c0 label: 16777215
Info (0xee1bb6c0): Tribe 0xf5164548: Bitmap 0x1: Mod-Bitmap 0x0: next (nil): label:
˓→16777215 attr 0xf3e57d30: withdraw 0

rcvd-path-id: 0 (0x0) advt-path-id: 0

Sent to:
VR 4: Group 20.0.0.100: Peer 20.0.0.100(0xf51a9690):

Attr: my ptr 0xf3e57d30: IPV4: ref 1: flags LOC_PREF_PRSNT GEN_LABEL : num asn 1: as-
˓→len: 1:

len 144: asp len 8: comm len 0

ext_comm len 0: cluster len 0; pmsi len 0; tot_uta len 0
CRC checksum = 0x2b35fcf4: asp ptr 0xf3e57db8: comm ptr 0xf3e57dc0
ext_comm ptr 0xf3e57dc0: cluster_ptr 0xf3e57dc0: pmsi_ptr 0xf3e57dc0: uta
˓→0xf3e57dc0

nexthop 0.0.0.0
prio_modified 0: fc_modified 0: fc 0: priority:0
med 0: loc pref 200: aggr ip 0.0.0.0: aggr as 0: origin 0
AIGP 0 0, 0 0
Orig ID 0.0.0.0: rt_pref 0

6.1. BGP 371

 VNS User Guide, Release 5.3.2

igp_type 0: igp-inst: 0: opaque-igp-data: 0x0: proto-data: 0x0

orig_ifid 0x0: vr_id 0: orig_rcvd_lbl: 0
global_if_id 0x0:
AS_SEQUENCE: 1: AS #s 300

Route: 20.0.1.0/24
my ptr 0xf514ed70: ro_head ptr 0xee1bbea0 label: 16777215
Info (0xee1bbea0): Tribe 0xf5165928: Bitmap 0x1: Mod-Bitmap 0x0: next 0xee1bba20:
˓→label: 16777215 attr 0xf3e57e60: withdraw 0

rcvd-path-id: 0 (0x0) advt-path-id: 0

Sent to:
VR 4: Group 20.0.0.3: Peer 20.0.0.3(0xf51ab7d0):

Attr: my ptr 0xf3e57e60: IPV4: ref 7: flags NULL: num asn 1: as-len: 1:
len 144: asp len 8: comm len 0
ext_comm len 0: cluster len 0; pmsi len 0; tot_uta len 0
CRC checksum = 0xb91081ab: asp ptr 0xf3e57ee8: comm ptr 0xf3e57ef0
ext_comm ptr 0xf3e57ef0: cluster_ptr 0xf3e57ef0: pmsi_ptr 0xf3e57ef0: uta
˓→0xf3e57ef0

nexthop 0.0.0.0
prio_modified 0: fc_modified 0: fc 0: priority:0
med 0: loc pref 0: aggr ip 0.0.0.0: aggr as 0: origin 2
AIGP 0 0, 0 0
Orig ID 0.0.0.0: rt_pref 0
igp_type 0: igp-inst: 0: opaque-igp-data: 0x0: proto-data: 0x0
orig_ifid 0x0: vr_id 0: orig_rcvd_lbl: 0
global_if_id 0x0:
AS_SEQUENCE: 1: AS #s 200

Info (0xee1bba20): Tribe 0xf5164548: Bitmap 0x1: Mod-Bitmap 0x0: next (nil): label:
˓→16777215 attr 0xf51ae9f8: withdraw 0

rcvd-path-id: 0 (0x0) advt-path-id: 0

Sent to:
VR 4: Group 20.0.0.100: Peer 20.0.0.100(0xf51a9690):

Attr: my ptr 0xf51ae9f8: IPV4: ref 7: flags LOC_PREF_PRSNT : num asn 0: as-len: 0:
len 136: asp len 0: comm len 0
ext_comm len 0: cluster len 0; pmsi len 0; tot_uta len 0
CRC checksum = 0xf9634e2e: asp ptr 0xf51aea80: comm ptr 0xf51aea80
ext_comm ptr 0xf51aea80: cluster_ptr 0xf51aea80: pmsi_ptr 0xf51aea80: uta
˓→0xf51aea80

nexthop 0.0.0.0
prio_modified 0: fc_modified 0: fc 0: priority:0
med 0: loc pref 200: aggr ip 0.0.0.0: aggr as 0: origin 2
AIGP 0 0, 0 0
Orig ID 0.0.0.0: rt_pref 0
igp_type 0: igp-inst: 0: opaque-igp-data: 0x0: proto-data: 0x0
orig_ifid 0x0: vr_id 0: orig_rcvd_lbl: 0
global_if_id 0x0:

Route: 20.0.2.0/24
my ptr 0xf514ec20: ro_head ptr 0xee1bc440 label: 16777215
Info (0xee1bc440): Tribe 0xf5165928: Bitmap 0x1: Mod-Bitmap 0x0: next 0xee1bb9f0:
˓→label: 16777215 attr 0xf3e57e60: withdraw 0

rcvd-path-id: 0 (0x0) advt-path-id: 0

Sent to:
VR 4: Group 20.0.0.3: Peer 20.0.0.3(0xf51ab7d0):

6.1. BGP 372

 VNS User Guide, Release 5.3.2

Attr: my ptr 0xf3e57e60: IPV4: ref 7: flags NULL: num asn 1: as-len: 1:
len 144: asp len 8: comm len 0
ext_comm len 0: cluster len 0; pmsi len 0; tot_uta len 0
CRC checksum = 0xb91081ab: asp ptr 0xf3e57ee8: comm ptr 0xf3e57ef0
ext_comm ptr 0xf3e57ef0: cluster_ptr 0xf3e57ef0: pmsi_ptr 0xf3e57ef0: uta
˓→0xf3e57ef0

nexthop 0.0.0.0
prio_modified 0: fc_modified 0: fc 0: priority:0
med 0: loc pref 0: aggr ip 0.0.0.0: aggr as 0: origin 2
AIGP 0 0, 0 0
Orig ID 0.0.0.0: rt_pref 0
igp_type 0: igp-inst: 0: opaque-igp-data: 0x0: proto-data: 0x0
orig_ifid 0x0: vr_id 0: orig_rcvd_lbl: 0
global_if_id 0x0:
AS_SEQUENCE: 1: AS #s 200

Info (0xee1bb9f0): Tribe 0xf5164548: Bitmap 0x1: Mod-Bitmap 0x0: next (nil): label:
˓→16777215 attr 0xf51ae9f8: withdraw 0

rcvd-path-id: 0 (0x0) advt-path-id: 0

Sent to:
VR 4: Group 20.0.0.100: Peer 20.0.0.100(0xf51a9690):

Attr: my ptr 0xf51ae9f8: IPV4: ref 7: flags LOC_PREF_PRSNT : num asn 0: as-len: 0:
len 136: asp len 0: comm len 0
ext_comm len 0: cluster len 0; pmsi len 0; tot_uta len 0
CRC checksum = 0xf9634e2e: asp ptr 0xf51aea80: comm ptr 0xf51aea80
ext_comm ptr 0xf51aea80: cluster_ptr 0xf51aea80: pmsi_ptr 0xf51aea80: uta
˓→0xf51aea80

nexthop 0.0.0.0
prio_modified 0: fc_modified 0: fc 0: priority:0
med 0: loc pref 200: aggr ip 0.0.0.0: aggr as 0: origin 2
AIGP 0 0, 0 0
Orig ID 0.0.0.0: rt_pref 0
igp_type 0: igp-inst: 0: opaque-igp-data: 0x0: proto-data: 0x0
orig_ifid 0x0: vr_id 0: orig_rcvd_lbl: 0
global_if_id 0x0:

Route: 20.0.3.0/24
my ptr 0xf514ece0: ro_head ptr 0xee1bc410 label: 16777215
Info (0xee1bc410): Tribe 0xf5165928: Bitmap 0x1: Mod-Bitmap 0x0: next 0xee1bb9c0:
˓→label: 16777215 attr 0xf3e57e60: withdraw 0

rcvd-path-id: 0 (0x0) advt-path-id: 0

Sent to:
VR 4: Group 20.0.0.3: Peer 20.0.0.3(0xf51ab7d0):

Attr: my ptr 0xf3e57e60: IPV4: ref 7: flags NULL: num asn 1: as-len: 1:
len 144: asp len 8: comm len 0
ext_comm len 0: cluster len 0; pmsi len 0; tot_uta len 0
CRC checksum = 0xb91081ab: asp ptr 0xf3e57ee8: comm ptr 0xf3e57ef0
ext_comm ptr 0xf3e57ef0: cluster_ptr 0xf3e57ef0: pmsi_ptr 0xf3e57ef0: uta
˓→0xf3e57ef0

nexthop 0.0.0.0
prio_modified 0: fc_modified 0: fc 0: priority:0
med 0: loc pref 0: aggr ip 0.0.0.0: aggr as 0: origin 2
AIGP 0 0, 0 0
Orig ID 0.0.0.0: rt_pref 0
igp_type 0: igp-inst: 0: opaque-igp-data: 0x0: proto-data: 0x0
orig_ifid 0x0: vr_id 0: orig_rcvd_lbl: 0

6.1. BGP 373

 VNS User Guide, Release 5.3.2

global_if_id 0x0:
AS_SEQUENCE: 1: AS #s 200

Info (0xee1bb9c0): Tribe 0xf5164548: Bitmap 0x1: Mod-Bitmap 0x0: next (nil): label:
˓→16777215 attr 0xf51ae9f8: withdraw 0

rcvd-path-id: 0 (0x0) advt-path-id: 0

Sent to:
VR 4: Group 20.0.0.100: Peer 20.0.0.100(0xf51a9690):

Attr: my ptr 0xf51ae9f8: IPV4: ref 7: flags LOC_PREF_PRSNT : num asn 0: as-len: 0:
len 136: asp len 0: comm len 0
ext_comm len 0: cluster len 0; pmsi len 0; tot_uta len 0
CRC checksum = 0xf9634e2e: asp ptr 0xf51aea80: comm ptr 0xf51aea80
ext_comm ptr 0xf51aea80: cluster_ptr 0xf51aea80: pmsi_ptr 0xf51aea80: uta
˓→0xf51aea80

nexthop 0.0.0.0
prio_modified 0: fc_modified 0: fc 0: priority:0
med 0: loc pref 200: aggr ip 0.0.0.0: aggr as 0: origin 2
AIGP 0 0, 0 0
Orig ID 0.0.0.0: rt_pref 0
igp_type 0: igp-inst: 0: opaque-igp-data: 0x0: proto-data: 0x0
orig_ifid 0x0: vr_id 0: orig_rcvd_lbl: 0
global_if_id 0x0:

Route: 20.0.4.0/24
my ptr 0xf514f3a0: ro_head ptr 0xee1bbc30 label: 16777215
Info (0xee1bbc30): Tribe 0xf5165928: Bitmap 0x1: Mod-Bitmap 0x0: next 0xee1bb990:
˓→label: 16777215 attr 0xf3e57e60: withdraw 0

rcvd-path-id: 0 (0x0) advt-path-id: 0

Sent to:
VR 4: Group 20.0.0.3: Peer 20.0.0.3(0xf51ab7d0):

Attr: my ptr 0xf3e57e60: IPV4: ref 7: flags NULL: num asn 1: as-len: 1:
len 144: asp len 8: comm len 0
ext_comm len 0: cluster len 0; pmsi len 0; tot_uta len 0
CRC checksum = 0xb91081ab: asp ptr 0xf3e57ee8: comm ptr 0xf3e57ef0
ext_comm ptr 0xf3e57ef0: cluster_ptr 0xf3e57ef0: pmsi_ptr 0xf3e57ef0: uta
˓→0xf3e57ef0

nexthop 0.0.0.0
prio_modified 0: fc_modified 0: fc 0: priority:0
med 0: loc pref 0: aggr ip 0.0.0.0: aggr as 0: origin 2
AIGP 0 0, 0 0
Orig ID 0.0.0.0: rt_pref 0
igp_type 0: igp-inst: 0: opaque-igp-data: 0x0: proto-data: 0x0
orig_ifid 0x0: vr_id 0: orig_rcvd_lbl: 0
global_if_id 0x0:
AS_SEQUENCE: 1: AS #s 200

Info (0xee1bb990): Tribe 0xf5164548: Bitmap 0x1: Mod-Bitmap 0x0: next (nil): label:
˓→16777215 attr 0xf51ae9f8: withdraw 0

rcvd-path-id: 0 (0x0) advt-path-id: 0

Sent to:
VR 4: Group 20.0.0.100: Peer 20.0.0.100(0xf51a9690):

Attr: my ptr 0xf51ae9f8: IPV4: ref 7: flags LOC_PREF_PRSNT : num asn 0: as-len: 0:
len 136: asp len 0: comm len 0
ext_comm len 0: cluster len 0; pmsi len 0; tot_uta len 0
CRC checksum = 0xf9634e2e: asp ptr 0xf51aea80: comm ptr 0xf51aea80

6.1. BGP 374

 VNS User Guide, Release 5.3.2

ext_comm ptr 0xf51aea80: cluster_ptr 0xf51aea80: pmsi_ptr 0xf51aea80: uta

˓→ 0xf51aea80
nexthop 0.0.0.0
prio_modified 0: fc_modified 0: fc 0: priority:0
med 0: loc pref 200: aggr ip 0.0.0.0: aggr as 0: origin 2
AIGP 0 0, 0 0
Orig ID 0.0.0.0: rt_pref 0
igp_type 0: igp-inst: 0: opaque-igp-data: 0x0: proto-data: 0x0
orig_ifid 0x0: vr_id 0: orig_rcvd_lbl: 0
global_if_id 0x0:

Route: 20.0.5.0/24
my ptr 0xf514ed10: ro_head ptr 0xee1bbb40 label: 16777215
Info (0xee1bbb40): Tribe 0xf5165928: Bitmap 0x1: Mod-Bitmap 0x0: next 0xee1bbba0:
˓→label: 16777215 attr 0xf3e57e60: withdraw 0

rcvd-path-id: 0 (0x0) advt-path-id: 0

Sent to:
VR 4: Group 20.0.0.3: Peer 20.0.0.3(0xf51ab7d0):

Attr: my ptr 0xf3e57e60: IPV4: ref 7: flags NULL: num asn 1: as-len: 1:
len 144: asp len 8: comm len 0
ext_comm len 0: cluster len 0; pmsi len 0; tot_uta len 0
CRC checksum = 0xb91081ab: asp ptr 0xf3e57ee8: comm ptr 0xf3e57ef0
ext_comm ptr 0xf3e57ef0: cluster_ptr 0xf3e57ef0: pmsi_ptr 0xf3e57ef0: uta
˓→0xf3e57ef0

nexthop 0.0.0.0
prio_modified 0: fc_modified 0: fc 0: priority:0
med 0: loc pref 0: aggr ip 0.0.0.0: aggr as 0: origin 2
AIGP 0 0, 0 0
Orig ID 0.0.0.0: rt_pref 0
igp_type 0: igp-inst: 0: opaque-igp-data: 0x0: proto-data: 0x0
orig_ifid 0x0: vr_id 0: orig_rcvd_lbl: 0
global_if_id 0x0:
AS_SEQUENCE: 1: AS #s 200

Info (0xee1bbba0): Tribe 0xf5164548: Bitmap 0x1: Mod-Bitmap 0x0: next (nil): label:
˓→16777215 attr 0xf51ae9f8: withdraw 0

rcvd-path-id: 0 (0x0) advt-path-id: 0

Sent to:
VR 4: Group 20.0.0.100: Peer 20.0.0.100(0xf51a9690):

Attr: my ptr 0xf51ae9f8: IPV4: ref 7: flags LOC_PREF_PRSNT : num asn 0: as-len: 0:
len 136: asp len 0: comm len 0
ext_comm len 0: cluster len 0; pmsi len 0; tot_uta len 0
CRC checksum = 0xf9634e2e: asp ptr 0xf51aea80: comm ptr 0xf51aea80
ext_comm ptr 0xf51aea80: cluster_ptr 0xf51aea80: pmsi_ptr 0xf51aea80: uta
˓→0xf51aea80

nexthop 0.0.0.0
prio_modified 0: fc_modified 0: fc 0: priority:0
med 0: loc pref 200: aggr ip 0.0.0.0: aggr as 0: origin 2
AIGP 0 0, 0 0
Orig ID 0.0.0.0: rt_pref 0
igp_type 0: igp-inst: 0: opaque-igp-data: 0x0: proto-data: 0x0
orig_ifid 0x0: vr_id 0: orig_rcvd_lbl: 0
global_if_id 0x0:

Route: 20.0.6.0/24
my ptr 0xf514eb60: ro_head ptr 0xee1bbb70 label: 16777215

6.1. BGP 375

 VNS User Guide, Release 5.3.2

Info (0xee1bbb70): Tribe 0xf5165928: Bitmap 0x1: Mod-Bitmap 0x0: next 0xee1bbbd0:
˓→label: 16777215 attr 0xf3e57e60: withdraw 0

rcvd-path-id: 0 (0x0) advt-path-id: 0

Sent to:
VR 4: Group 20.0.0.3: Peer 20.0.0.3(0xf51ab7d0):

Attr: my ptr 0xf3e57e60: IPV4: ref 7: flags NULL: num asn 1: as-len: 1:
len 144: asp len 8: comm len 0
ext_comm len 0: cluster len 0; pmsi len 0; tot_uta len 0
CRC checksum = 0xb91081ab: asp ptr 0xf3e57ee8: comm ptr 0xf3e57ef0
ext_comm ptr 0xf3e57ef0: cluster_ptr 0xf3e57ef0: pmsi_ptr 0xf3e57ef0: uta
˓→0xf3e57ef0

nexthop 0.0.0.0
prio_modified 0: fc_modified 0: fc 0: priority:0
med 0: loc pref 0: aggr ip 0.0.0.0: aggr as 0: origin 2
AIGP 0 0, 0 0
Orig ID 0.0.0.0: rt_pref 0
igp_type 0: igp-inst: 0: opaque-igp-data: 0x0: proto-data: 0x0
orig_ifid 0x0: vr_id 0: orig_rcvd_lbl: 0
global_if_id 0x0:
AS_SEQUENCE: 1: AS #s 200

Info (0xee1bbbd0): Tribe 0xf5164548: Bitmap 0x1: Mod-Bitmap 0x0: next (nil): label:
˓→16777215 attr 0xf51ae9f8: withdraw 0

rcvd-path-id: 0 (0x0) advt-path-id: 0

Sent to:
VR 4: Group 20.0.0.100: Peer 20.0.0.100(0xf51a9690):

Attr: my ptr 0xf51ae9f8: IPV4: ref 7: flags LOC_PREF_PRSNT : num asn 0: as-len: 0:
len 136: asp len 0: comm len 0
ext_comm len 0: cluster len 0; pmsi len 0; tot_uta len 0
CRC checksum = 0xf9634e2e: asp ptr 0xf51aea80: comm ptr 0xf51aea80
ext_comm ptr 0xf51aea80: cluster_ptr 0xf51aea80: pmsi_ptr 0xf51aea80: uta
˓→0xf51aea80

nexthop 0.0.0.0
prio_modified 0: fc_modified 0: fc 0: priority:0
med 0: loc pref 200: aggr ip 0.0.0.0: aggr as 0: origin 2
AIGP 0 0, 0 0
Orig ID 0.0.0.0: rt_pref 0
igp_type 0: igp-inst: 0: opaque-igp-data: 0x0: proto-data: 0x0
orig_ifid 0x0: vr_id 0: orig_rcvd_lbl: 0
global_if_id 0x0:

Route: 20.0.7.0/24
my ptr 0xf514f7c0: ro_head ptr 0xee1bbf60 label: 16777215
Info (0xee1bbf60): Tribe 0xf5165928: Bitmap 0x1: Mod-Bitmap 0x0: next 0xee1bb960:
˓→label: 16777215 attr 0xf3e57e60: withdraw 0

rcvd-path-id: 0 (0x0) advt-path-id: 0

Sent to:
VR 4: Group 20.0.0.3: Peer 20.0.0.3(0xf51ab7d0):

Attr: my ptr 0xf3e57e60: IPV4: ref 7: flags NULL: num asn 1: as-len: 1:
len 144: asp len 8: comm len 0
ext_comm len 0: cluster len 0; pmsi len 0; tot_uta len 0
CRC checksum = 0xb91081ab: asp ptr 0xf3e57ee8: comm ptr 0xf3e57ef0
ext_comm ptr 0xf3e57ef0: cluster_ptr 0xf3e57ef0: pmsi_ptr 0xf3e57ef0: uta
˓→0xf3e57ef0

nexthop 0.0.0.0

6.1. BGP 376

 VNS User Guide, Release 5.3.2

prio_modified 0: fc_modified 0: fc 0: priority:0

med 0: loc pref 0: aggr ip 0.0.0.0: aggr as 0: origin 2
AIGP 0 0, 0 0
Orig ID 0.0.0.0: rt_pref 0
igp_type 0: igp-inst: 0: opaque-igp-data: 0x0: proto-data: 0x0
orig_ifid 0x0: vr_id 0: orig_rcvd_lbl: 0
global_if_id 0x0:
AS_SEQUENCE: 1: AS #s 200

Info (0xee1bb960): Tribe 0xf5164548: Bitmap 0x1: Mod-Bitmap 0x0: next (nil): label:
˓→16777215 attr 0xf51ae9f8: withdraw 0

rcvd-path-id: 0 (0x0) advt-path-id: 0

Sent to:
VR 4: Group 20.0.0.100: Peer 20.0.0.100(0xf51a9690):

Attr: my ptr 0xf51ae9f8: IPV4: ref 7: flags LOC_PREF_PRSNT : num asn 0: as-len: 0:
len 136: asp len 0: comm len 0
ext_comm len 0: cluster len 0; pmsi len 0; tot_uta len 0
CRC checksum = 0xf9634e2e: asp ptr 0xf51aea80: comm ptr 0xf51aea80
ext_comm ptr 0xf51aea80: cluster_ptr 0xf51aea80: pmsi_ptr 0xf51aea80: uta
˓→0xf51aea80

nexthop 0.0.0.0
prio_modified 0: fc_modified 0: fc 0: priority:0
med 0: loc pref 200: aggr ip 0.0.0.0: aggr as 0: origin 2
AIGP 0 0, 0 0
Orig ID 0.0.0.0: rt_pref 0
igp_type 0: igp-inst: 0: opaque-igp-data: 0x0: proto-data: 0x0
orig_ifid 0x0: vr_id 0: orig_rcvd_lbl: 0
global_if_id 0x0:

6.1.10 VIF RTM CLI Commands

The CLI commands in this section can be used to assess whether an underlay is reachable from an NSG via a VIF
route.
The following CLI commands are available:
• nuage-bgp-show vifs
• nuage-bgp-show vifs 2
• nuage-bgp-show underlays
• nuage-bgp-show underlays 2
The nuage-bgp-show vifs command lists the available VIF routes and indicates whether the remote VIF is
reachable. A value of 1 for the Reach parameter indicates that the VIF route is reachable.

vsc# nuage-bgp-show vifs

My Vif ID: 3.70.252.42 Vif count 3

Me VifID UndCnt isUBR Reach Use Rtm

==================================================
* 3.70.252.42 1 0 1 0 1
6.15.154.189 3 1 1 1 1
37.178.54.126 2 0 1 0 1
--------------------------------------------------

6.1. BGP 377

 VNS User Guide, Release 5.3.2

The nuage-bgp-show vifs 2 command lists the available VIF routes in greater detail than the
nuage-bgp-show vifs command. In addition to indicating the reachability of each VIF route, the command
also displays the associated underlay for each NSG. A value of 1 for the Reach parameter indicates that the VIF route
is reachable.

vsc# nuage-bgp-show vifs 2

My Vif ID: 3.70.252.42 Vif count 3

Me VifID UndCnt isUBR Reach Use Rtm

==============================================================
* 3.70.252.42 1 0 1 0 1
* Underlay 53600 Reachable 1 UBR_Count 2 Nsg_Count 2
--------------------------------------------------------------
6.15.154.189 3 1 1 1 1
Underlay 0 Reachable 1 UBR_Count 2 Nsg_Count 0
Underlay 30788 Reachable 1 UBR_Count 2 Nsg_Count 2
* Underlay 53600 Reachable 1 UBR_Count 2 Nsg_Count 2
--------------------------------------------------------------
37.178.54.126 2 0 1 0 1
Underlay 30788 Reachable 1 UBR_Count 2 Nsg_Count 2
* Underlay 53600 Reachable 1 UBR_Count 2 Nsg_Count 2
--------------------------------------------------------------

The nuage-bgp-show underlays command lists the underlays that are known to the NSG and indicates whether
or not they are reachable. A value of 1 for the Reach parameter indicates that the underlay is reachable.

vsc# nuage-bgp-show underlays

Underlay count 3

Me Underlay UbrCnt NsgCnt State Reach SndUbrOnly

==================================================
0 2 0 Up 1 0
1863 0 0 Up 0 0
30788 2 2 Up 1 0
--------------------------------------------------

The nuage-bgp-show underlays 2 command lists the underlays that are known to the NSG in greater detail
that the nuage-bgp-show underlays command. In addition to indicating the reachability of the known under-
lays, the command also lists all associated NSGs and NSG-UBRs for each underlay. A value of 1 for the Reach
parameter indicates that the underlay is reachable.

vsc# nuage-bgp-show underlays 2

Underlay count 3

Me Underlay UbrCnt NsgCnt State Reach SndUbrOnly

===================================================
0 2 0 Up 1 0
Vif 202.130.26.72 isUBR 1 Reachable 1
Vif 6.15.154.189 isUBR 1 Reachable 1
---------------------------------------------------
1863 0 0 Up 0 0
---------------------------------------------------
30788 2 2 Up 1 0
Vif 37.178.54.126 isUBR 0 Reachable 1
Vif 102.113.229.65 isUBR 0 Reachable 1
Vif 202.130.26.72 isUBR 1 Reachable 1
Vif 6.15.154.189 isUBR 1 Reachable 1
---------------------------------------------------

6.1. BGP 378

 VNS User Guide, Release 5.3.2

6.2 OSPF

• OSPF Overview (page 379)

– OSPF Areas (page 380)

* Backbone Area (page 380)

* Normal Area (page 380)
* Stub Area (page 380)
* Not So Stubby Area (page 381)
• OSPF Configuration (page 381)
– OSPF Instance (page 381)
– OSPF Area (page 382)
– OSPF Interface (page 383)
– Routing Policy (page 383)
– Policy Object Groups (page 384)
– NSG Routing Policy Binding (page 384)

* Route Redistribution (page 384)

• Provisioning Workflow (page 385)
– Enable Routing and Configure an AS (page 385)
– Create and Instantiate an L3 Domain (page 385)
– Create Routing Policies (page 386)

* Create Routing Policies at the Domain Level (page 386)

* Create Routing Policies at the Organization Level (page 386)
– Create an OSPF Instance (page 386)
– Create an OSPF Area (page 386)
– Create an OSPF Interface (page 387)
– Create Policy Object Groups (page 387)
– Create NSG Routing Policy Bindings (page 387)
• OSPF Routing Policy Blob (page 388)
– OSPF Routing Scenarios (page 389)
• OSPF CLI Samples (page 389)

6.2.1 OSPF Overview

The Nuage VNS solution supports Open Shortest Path First version 2 (OSPFv2) on NSG V-Ports to enable the ex-
change and redistribution of route topology information into either a network overlay or an IPVPN underlay. OSPF is
a link state protocol that allows routers to exchange route topology information so that routers can calculate the lowest
cost routes before populating routing tables.

6.2. OSPF 379

 VNS User Guide, Release 5.3.2

Link state advertisements (LSAs) are the messages that allow routers configured with OSPF to exchange route topol-
ogy information with OSPF neighbors. Each LSA type is handled differently. Some types are flooded only within an
OSPF area, and some types are generated or forwarded only by Area border routers (ABRs) depending on area type.
These limitations on LSA communication allow for more scalability network-wide.

Note: In the initial release of OSPF for VNS, “Super Backbone” is not supported. As such, the NSG will not function
as an ABR towards the overlay, and LSAs are not carried transparently over the overlay. This means that each area
attached to an NSG is essentially isolated from an OSPF perspective, i.e. there is no backbone area in the overlay.
Routes being advertised into or out of the area by the NSG are therefore “redistributed” and the NSG functions as an
ASBR i.e. all routes coming from the overlay will be redistributed into the area as type 5 or type 7 LSAs depending on
configuration. Area types and configurations are still applicable for LSA transmission between locally defined areas
on a given NSG.

OSPF Areas

OSPF relies on the concept of logical areas. Areas allow topology information between areas to be hidden while still
providing reachability. Each router in the area shares the same LSA database and routing tables, which simplifies the
network topology and helps to optimize the route calculation algorithm. All areas in the OSPF network must connect
to the backbone area (area 0). Area border routers (ABRs) are used to connect two OSPF areas. ABRs are used to
provide route summarization to neighboring areas. Autonomous system border routers (ASBRs) are used to connect
the OSPF network to another AS using a different routing protocol.
OSPF area types are defined by how they handle different LSAs. VNS supports the OSPF area types described below.

Backbone Area

A backbone area (area 0) is a normal area which has been configured as the central area which connects to all other
areas in the OSPF domain. The backbone area is responsible for distributing route topology information to connected
non-backbone areas. All other OSPF areas in the network must be connected to the backbone area. The backbone area
usually includes several routers configured as ABRs to connect with other areas in the network.

Normal Area

A normal area, by default, allows all LSA types. Normal areas allow type 3 LSAs — representing routes learned by
the ABR from a connected area — to be summarized and sent to and from the backbone area. Normal areas also allow
type 4 LSAs for ASBR-learned route summaries, as well as type 5 LSAs for external routes. This means that an NSG
can be configured as an ASBR and redistribute external routes into the normal area as type 5 LSAs. By default, an
NSG is always an ASBR and may optionally be an ABR depending on configuration.
The Area Range parameters specify a range of contiguous routes to be aggregated or suppressed within the area.
In a normal area, an aggregation and suppression Area Range can be specified for type 3 LSAs. When routes are
suppressed, they are not advertised into the overlay or underlay.

Stub Area

A stub area is an area that does not allow route advertisements from outside the AS. An ABR that is connected to a stub
area will not forward type 5 LSAs into the area, consequently the stub area will not receive any external routes from
the backbone area. Stub areas allow type 3 LSAs for internal route summarization, and by default, will accept Type 3
LSAs from the backbone area. All external routing is based on one default route (0.0.0.0) generated by the ABR. A
stub area must contain at least one ABR connecting to the backbone area. A stub area cannot contain an ASBR. Stub

6.2. OSPF 380

 VNS User Guide, Release 5.3.2

areas are useful when a default route is sufficient for all external networks and type 5 LSAs from the backbone area
are not needed. If the area needs to allow an ASBR to inject AS-external routes then a Not So Stubby Area (NSSA)
should be used instead.
The Summaries Enabled parameter enables or disables type 3 LSAs. When type 3 LSAs are disabled in a NSSA, it
behaves as a Totally Stubby Area (TSA). The ABR generates Type 3 LSAs towards the backbone area representing
the stub area routes, but will not advertise any Type 3 LSAs into the area. In a TSA, the default route generated by the
ABR is the only route used for traffic flowing outside the area.
The Area Range parameters specify a range of contiguous routes to be aggregated or suppressed within the area. In a
stub area, an aggregation and suppression Area Range can be specified for type 3 LSAs. When routes are suppressed,
they are not advertised into the overlay or underlay. If the Summaries Enabled parameter is disabled, area ranges for
type 3 LSAs cannot be configured.

Not So Stubby Area

A Not So Stubby Area (NSSA) is a stub area that also supports type 7 LSAs for external route forwarding. NSSAs
cannot receive route advertisements from outside the AS arriving via the backbone area. Routers in the NSSA can
be an ASBR and advertise external routes as a type 7 LSA. The ABR translates these advertisements to type 4 and 5
LSAs for redistribution the backbone area. An NSSA must contain at least one ABR connecting to the backbone area.
NSSAs provide control over external routes from the backbone area. They also allow an external ASBR in the area to
redirect type 7 LSAs into the backbone area.
The generation of a default route can be enabled by configuring the Default Originate Option. The Summaries
Enabled parameter enables or disables type 3 LSAs. When type 3 LSAs are disabled in a NSSA, it behaves as a
Totally Not So Stubby Area (TNSSA). If default routes are enabled, and type 3 LSAs are disabled, no overlay or
underlay routes are redistributed into the NSSA — only the default 0.0.0.0/0 route. The default route is usually a type
3 LSA, but it can be configured as a type 7 LSA if required. The Default Originate Option parameter is disabled by
default.
If the Redistribute External parameter is enabled for an NSSA, the NSG injects routes learned from the overlay or
underlay as type 7 LSAs into the area.
The Area Range parameters specify a range of contiguous routes to be aggregated or suppressed within the area. In an
NSSA, an aggregation and suppression Area Range can be specified for each of type 3 LSAs and type 7 LSAs. When
routes are suppressed, they are not advertised into the overlay or underlay. If the Summaries Enabled parameter is
disabled, area ranges for type 3 LSAs cannot be configured.

6.2.2 OSPF Configuration

The following network objects must be configured as part of the OSPF workflow.

OSPF Instance

The OSPF instance is the highest hierarchical OSPF configuration object in a domain. The OSPF instance assigns
global import and export routing policies for OSPF traffic in the domain. Routing policies can also be defined and
assigned locally to an NSG or NSG group by using an NSG routing policy binding, in which case the policies override
the globally defined policies. Only one OSPF instance can be configured per domain.
Configure the following parameters for an OSPF Instance:
• Name: This parameter defines a name for the OSPF instance.
• Description: This parameter defines a description for the OSPF instance.
• Import Routing Policy: This parameter assigns an import routing policy to the OSPF instance.

6.2. OSPF 381

 VNS User Guide, Release 5.3.2

• Export Routing Policy: This parameter assigns an export routing policy to the OSPF instance.
• Preference: This parameter defines a routing preference value for OSPF internal routes.
• External Preference: This parameter defines a routing preference value for OSPF external routes.
• Export Limit
• Enable Route Redistribution to Overlay: This parameter enables route redistribution globally. When the
parameter is enabled, the NSG redistributes OSPF routes into the overlay.

OSPF Area

The OSPF area is configured at the OSPF instance level. Multiple OSPF areas can be configured per OSPF instance.
Configure the following parameters for an OSPF area:
• Area ID: This parameter identifies the OSPF area within the instance.
• Description: This parameter defines the description for the OSPF area.
• Area Type: This parameter defines the area type; choose Normal, NSSA, or Stub.
• Default Originate Option: This parameter enables the creation of a default route by the ABR or ASBR. Specify
Type 3 or Type 7 LSAs.
• Suppress Area Range/Aggregate Area Range: These parameters define address ranges on the ABR for route
suppression or aggregation. For all area types, area ranges can be configured for type 3 LSAs. For NSSAs, area
ranges can also be configured for type 7 LSAs using the Suppress Area Range NSSA/Aggregate Area Range
NSSA parameters.
• Summaries Enabled: This parameter enables or disables type 3 LSAs in the area. This parameter is applicable
only to NSSAs and stub areas.
• Enable Redistribute External: This parameter allows the NSG to inject routes learned from the overlay or
underlay as type 7 LSAs into the area. This parameter is applicable only to NSSAs.
• Default Metric: This parameter defines a route cost metric for routes advertised in the area. This parameter is
applicable only to stub areas.

Note: Once the area is created, the Area ID parameter cannot be configured. To change the area ID, delete the area
and recreate it. When an area is deleted, all associated OSPF interfaces are automatically deleted.

Note: Multiple areas with the same Area ID and Area Type is supported. The different areas can have different
attributes. This means that an area could be configured only once and used as a sort of template to configure the same
area accross multple NSGs/domains. Despite the areas using the same Area ID, however, it is not the same area.

In the case where two separate NSGs need to function as an ASBR/ABR for two separate areas but the areas have the
same area ID, but different area types, multiple copies of the configured area may need to be created.
For example, to configure Area 100 with type Normal on NSG 1 and type Stub on NSG 2, two copies of Area 100
must be created with interfaces for each NSG:
Area ID: 100 Area Type: Normal OSPF interface 1 Subnet: 1 NSG 1
Area ID: 100 Area Type: Stub OSPF interface 1 Subnet: 2 NSG 2

6.2. OSPF 382

 VNS User Guide, Release 5.3.2

OSPF Interface

The OSPF interface represents the connection of a router to the OSPF network. The OSPF interface defines the
protocol metrics and security parameters for OSPF traffic on a V-Port on the specified subnet. An OSPF interface can
exist in only one OSPF area.
Configure the following parameters for an OSPF interface:
• Name: This parameter defines a name for the OSPF interface.
• Description: This parameter defines a description for the OSPF interface.
• Hello Interval: This parameter defines the time interval, in seconds, between OSPF hello messages sent on the
OSPF instance. Specify a lower value to ensure that link failures are detected sooner.
• Dead Interval: This parameter defines the time, in seconds, that the OSPF interface waits after failing to receive
an OSPF hello message before declaring the neighbor interface down.
• Metric: This parameter defines a route cost metric for routes advertised to the OSPF interface. This route cost
metric is preferred over a metric defined at the OSPF area level.
• Admin State: This parameter turns the OSPF interface up or down.
• Passive Enabled: This parameter enables passive mode on the OSPF interface. In passive mode, the OSPF
interface does not forward OSPF packets.
• Subnet: This parameter assigns the OSPF interface to a subnet. This is effectively how an NSG vPort is bound
to an area.
• Interface Type: This parameter specifies whether the OSPF interface is either Broadcast or Point-to-Point.
Specify Point-to-Point to eliminate the broadcast traffic on the interface and reduce overhead.
• MTU: This parameter defines the OSPF packet size on the OSPF interface.
• Priority: This parameter defines a priority value for the OSPF interface. If the interface is configured as a
broadcast interface, the priority value determines which interface is elected as a designated router on the subnet.
• Key Type: This parameter defines the key type on the OSPF interface. Choose Hash Key, Hash 2 Key, or Key.
• Authentication Type: This parameter enables authentication and defines the authentication type on the OSPF
interface. Choose Message Digest, Password, or None
• Authentication Digest Key: This parameter defines a password when password authentication is enabled on
the OSPF interface.
• Message Digest Key: This parameter defines an MD5 key when message digest authentication is enabled on
the OSPF interface.
• Message Digest Key ID: This parameter defines a unique MD5 key ID when message digest authentication is
enabled on the OSPF interface.

Routing Policy

A routing policy allows control over route import and export behavior for OSPF traffic. Route redistribution can be
configured such that certain routes are used exclusively, or preferred over other routes based on defined metrics. For
example, a policy can be configured such that routes are redistributed with a preference for the underlay instead of the
BGP overlay.
Routing policies can be configured at the organization or domain level. They can be applied at either the OSPF instance
or NSG level. Routing policies assigned at the NSG level are preferred over policies assigned at the instance level.
By default, all BGP routes in the domain are picked up and advertised by OSPF. This behavior can be prevented by
configuring a routing policy to drop all BGP routes.

6.2. OSPF 383

 VNS User Guide, Release 5.3.2

Routing policy definitions use the blob model. See the OSPF Routing Policy Blob (page 388) for more information
about OSPF blobs.
Configure the following parameters for a routing policy:
• Name: This parameter defines a name for the routing policy.
• Description: This parameter defines a description for the routing policy.
• Default Route Action: This parameter defines a default action for received routes. Specify Accept or Reject.
• Policy Definition: This parameter defines the metrics and actions for OSPF traffic. Specify an OSPF routing
policy blob in the correct format.

Note: Default actions in the blob take precedence over the one configured with the Default Route Action parameter.

Policy Object Groups

The policy object group is used to assign NSGs to a group for routing policy entries. Most use cases would require
only one NSG for a routing policy entry, but policy object groups allow multiple NSGs in an entry.
Configure the following parameters for a policy object group:
• Name: This parameter defines a name for the policy object group.
• Description: This parameter defines a description for the policy object group.

NSG Routing Policy Binding

The NSG routing policy binding is used to assign routing policies to an NSG or group of NSGs as defined by the
routing policy group. The local routing policies assigned to an NSG in a policy binding are preferred over global
routing policies defined at the OSPF instance level.
If routing policies are configured at the domain level, they are automatically inherited locally.

Route Redistribution

OSPF route redistribution into the overlay can be enabled locally by configuring the Export to Overlay parameter.
When the parameter is enabled, NSGs in the policy object group redistribute OSPF routes into the overlay. OSPF route
redistribution from the overlay can be enabled locally by creating an export routing policy. By default, the Export to
Overlay parameter is disabled and no routes are redistributed to or from the overlay.
OSPF route redistribution into the underlay can be configured locally by creating export routing policies. By default,
all routes from a BGP underlay peer are not exported into OSPF, whereas all routes from OSPF are redistributed to the
BGP underlay peer.
OSPF route redistribution into the overlay can be configured globally by configuring the Enable Route Redistribution
to Overlay parameter on the OSPF instance. This setting is automatically inherited locally to all NSG policy bindings.
Configure the following parameters for an NSG routing policy binding:
• Name: This parameter defines a name for the policy binding.
• Description: This parameter defines a description for the policy binding.
• Export to Overlay: This parameter enables route redistribution locally. When the parameter is enabled, the
NSG redistributes OSPF routes into the overlay. When the parameter is set to Inherited, it inherits whatever
value is selected at the OSPF Instance level.

6.2. OSPF 384

 VNS User Guide, Release 5.3.2

• Import Routing Policy: This parameter assigns an import routing policy to the policy binding.
• Export Routing Policy: This parameter assigns an export routing policy to the policy binding.
• Policy Group Binding: This parameter assigns a policy group binding (with an associated NSG or group of
NSGs) to the policy binding.

6.2.3 Provisioning Workflow

Perform the following steps to configure OSPF.

Enable Routing and Configure an AS

Note: CSPRoot privileges are required to configure an organization profile.

1. Click on the Platform Configuration button.

2. Under the Organization Policies tab, click on the Organization Profiles button.

3. Click on the button to create a new organization profile, or right-click an existing profile and choose Edit.
4. Enable the Enable Routing Protocols parameter.

Note: Unless the Enable Routing Protocols parameter is enabled, OSPF and BGP functionality is not available. If
routing is disabled while OSPF configurations already exist in the VSD, the configurations are not lost, they are just
hidden in the UI. If routing is enabled again later, the previously configured OSPF configurations are restored.

5. Complete other configuration, as required, and click Update or Create.

Note: Organization profiles that are currently in use cannot be edited.

5. From the Organizations panel, click on the button to create a new organization.
6. Configure the Name, Description, Local AS, and Administrator Password parameters.

Note: The Local AS parameter is required when routing is enabled in an organization.

7. Assign the organization profile with routing enabled and click Create.

Create and Instantiate an L3 Domain

8. From the configured organization, click on the Network tab.

9. Click on the L3 Domains button and click on the button to create a new domain template.
10. Configure the Name and Description parameters and click Create.
11. Right-click on the domain template and choose Instantiate.
12. Configure the Name and Description parameters and click Instantiate.

6.2. OSPF 385

 VNS User Guide, Release 5.3.2

Create Routing Policies

Create Routing Policies at the Domain Level

13. From the right-hand information panel of the selected domain, click on the Routing Policies button.

14. Click on the Infrastructure tab and click on the Routing Policies button.
15. Configure the Name, Description, Default Route Action, and Policy Definition parameters.
Routing policy definitions use the blob model. See the OSPF Routing Policy Blob (page 388) for more informa-
tion about OSPF blobs.

Note: Default actions in the blob take precedence over the one configured with the Default Route Action parameter.

16. Click Create.

Create Routing Policies at the Organization Level

Note: Enterprise administrator privileges are required to configure routing policies at the organization level.

17. Click on the Infrastructure tab and click on the Routing Policies button.

18. Click on the button to create a new routing policy.

19. Configure the parameters as described at the domain level.

Create an OSPF Instance

Note: The OSPF tab is displayed only if routing protocols are enabled on the domain. If routing protocols are
disabled, the VSD database retains all previously configured OSPF parameters.

20. From the newly created domain, click on the OSPF tab and click on the OSPF Instances button.

21. Click on the button to create a new OSPF Instance.

22. Assign routing policies for the Import Routing Policy and Export Routing Policy parameters.
23. Configure the Name, Description, Preference, External Preference, Export Limit, and Enable Route De-
stribution to Overlay parameters and click Create.

Create an OSPF Area

24. Click on the newly created domain and click on the button to create a new OSPF area.
25. Configure the Area ID, Description, Area Type, Default Originate Option, Enable Summaries, Default
Metric, and Enable Redistribute External parameters.
26. Assign area ranges for the Suppress Area Range, Aggregate Area Range, Suppress Area Range NSSA, and
Aggregate Area Range NSSA parameters and click Create.

6.2. OSPF 386

 VNS User Guide, Release 5.3.2

Note: Some parameters are available only for certain area types. See OSPF Area (page 382) for more information on
OSPF area parameters.

Create an OSPF Interface

27. Click on the newly created area and click on the button to create a new OSPF interface.
28. Configure the Name and Description parameters.
29. From the Basic tab, configure the Hello Interval, Dead Interval, Metric, Admin State, and Passive Enabled
parameters.
30. Assign a Subnet to the OSPF interface.
31. Click on the Advanced tab.
32. Configure the Interface Type, MTU, Priority, Key Type, Authentication Type, Authentication Digest Key,
Message Digest Key, and Message Digest Key ID parameters and click Create.

Note: The Message Digest Key and Message Digest Key ID parameters are configurable only if Message Digest is
specified for the Key Type parameter.

Note: The Authentication Digest Key parameter is configurable only if Password is specified for the Key Type
parameter.

Create Policy Object Groups

Note: The remainder of this workflow assumes NSGs have been discovered into the domain.

33. From the Networks tab, click on the Policy Object Groups button.

34. Click on the button and configure the Name and Description parameters. Click Create.

35. Select the policy from the Policy Object Groups panel and click on the button.
36. Select an NSG to add to the Policy Object Group and click Select.

Create NSG Routing Policy Bindings

37. Return to the OSPF tab of the selected domain.

38. Click on the NSG Routing Policy Bindings button and click on the button to create a new binding.
39. Configure the Name, Description, and Export to Overlay parameters.
40. Assign routing policies for the Import Routing Policy and Export Routing Policy.
41. Assign a Policy Group Binding and click Create.

6.2. OSPF 387

 VNS User Guide, Release 5.3.2

6.2.4 OSPF Routing Policy Blob

A routing policy blob is an opaque object based on an extensible OpenConfig schema whereby configuration elements
are defined as part of an XML document. Define these elements in order to configure OSPF policies on the NSG to
control route import and export behavior.
This section shows the OSPF routing policy blob format and provides some examples of blob usage.
<routing-policy xmlns="alu:nuage:bgp:routing:policy">
<policy-definition>
<statements>
<statement>
<name>entry_1</name>
<conditions>
<match-prefix-set>
<prefix-set>prefix_list_1</prefix-set>
</match-prefix-set>
<match-neighbor-set>
<neighbor-set>neighbor_list_1</neighbor-set>
</match-neighbor-set>
<match-community-set>
<community-set>comm_1</community-set>
</match-community-set>
<match-as-path-set>
<as-path-set>as_path_1</as-path-set>
</match-as-path-set>
<install-protocol-eq>OSPF</install-protocol-eq>
<match-ospf-area>area-id</match-ospf-area>
<match-route-tag>Uint32</match-route-tag>
<match-ospf-type-metric>1 OR 2</match-ospf-type-metric>
</conditions>
<actions>
<!-- action can be either accept or reject, not both -->
<accept-route-set>
<accept-route />
<as-path-set>as_path_2</as-path-set>
<community-set>comm_2</community-set>
<local-preference>200</local-preference>
<metric>
<!-- select either of add/subtract/set options below -->
<add>10</add>
<subtract>10</subtract>
<set>110</set>
</metric>
<next-hop>10.10.1.1</next-hop>
<next-hop-self>true</next-hop-self>
<preference>100</preference>
</accept-route-set>
<!-- action can be either accept or reject, not both -->
<reject-route />
<match-ospf-tag>Uint32</match-ospf-tag>
<match-ospf-type>1 OR 2</match-ospf-type>
</actions>
</statement>
<statement>
...
</statement>
</statements>
<default-action>

6.2. OSPF 388

 VNS User Guide, Release 5.3.2

<accept-route-set>
<accept-route />
<community-set>comm_1</community-set>
<local-preference>200</local-preference>
</accept-route-set>
</default-action>
</policy-definition>
</routing-policy>

OSPF Routing Scenarios

Export from Local OSPF Import to Local OSPF

Overlay Not enabled by default. Enable Export to Not enabled by default. Use a routing policy to
Redistribu- Overlay to send local OSPF routes to the import routes from the overlay to the local
tion overlay. OSPF.
Underlay All routes are distributed from OSPF to the Not enabled by default. Use a routing policy to
Redistribu- underlay by default. Use a routing policy to distribute routes from the underlay to OSPF.
tion disable.

6.2.5 OSPF CLI Samples

This section lists samples of the CLI show commands available for OSPF.

vsc# tools vswitch 10.15.1.254 command "nuage-ospf-show help"

usage: nuage-ospf-show [-h] COMMAND [ARG [ARG ...]]

Nuage OSPF Show Commands

optional arguments:
-h, --help show this help message and exit

Commands Description Args:

-------------------------------------------------------------------------
config-detail Show applied ospf config for specific vrf DomainVrfId
config-all Show applied ospf config for all vrf None
status Show ospf status for specific vrf DomainVrfId
neighbor Show ospf neighbor for specific vrf DomainVrfId
neighbor-detail Show ospf neighbor for specific vrf DomainVrfId
routes Show ospf routes for specific vrf DomainVrfId
routes-detail Show ospf routes for specific vrf DomainVrfId
area Show ospf interfaces for specific vrf DomainVrfId
interface Show ospf interfaces for specific vrf DomainVrfId
database Show ospf database for specific vrf DomainVrfId
database-detail Show ospf database for specific vrf DomainVrfId
leaked-routes Show ospf leaked routes for specific vrf DomainVrfId
stats Show ospf statistics for specific vrf DomainVrfId
clear-stats Clear ospf statistics for specific vrf DomainVrfId
help Print help string None

COMMAND Command to use.

ARG Arguments to COMMAND.

##################################

6.2. OSPF 389

 VNS User Guide, Release 5.3.2

vsc# tools vswitch 10.15.1.254 command "nuage-ospf-show config-all"

===========================================
OSPF Config Vrf: 19999 RTM Vrf: 3
===========================================

===========================================
OSPF Config Vrf: 125668019 RTM Vrf: 2
===========================================
router-id 64.152.193.19
area 0.0.0.0
interface "Int-125668019-1657287575"
no shutdown
exit
interface "Int-125668019-602706402"
no shutdown
exit
exit
area 0.0.0.1
stub
exit
interface "Int-125668019-1696182375"
no shutdown
exit
exit

###################################

vsc# tools vswitch 10.15.1.254 command "nuage-ospf-show config-detail 125668019"

router-id 64.152.193.19
no super-backbone
no vpn-domain
vpn-tag 0
reference-bandwidth 100000000
compatible-rfc1583
no external-db-overflow
no overload-include-stub
no overload
no overload-on-boot
timers
no spf-wait
no lsa-generate
no lsa-arrival
exit
preference 10
external-preference 150
no export-limit
no export
no graceful-restart
no ignore-dn-bit
no suppress-dn-bit
no import
no loopfree-alternate
area 0.0.0.0
no loopfree-alternate-exclude
blackhole-aggregate
interface "Int-125668019-1657287575"
no passive

6.2. OSPF 390

 VNS User Guide, Release 5.3.2

interface-type broadcast
advertise-subnet
priority 1
hello-interval 10
dead-interval 40
retransmit-interval 5
transit-delay 1
no mtu
no metric
no authentication-type
no authentication-key
no bfd-enable
no loopfree-alternate-exclude
no shutdown
exit
interface "Int-125668019-602706402"
no passive
interface-type broadcast
advertise-subnet
priority 1
hello-interval 10
dead-interval 40
retransmit-interval 5
transit-delay 1
no mtu
no metric
no authentication-type
no authentication-key
no bfd-enable
no loopfree-alternate-exclude
no shutdown
exit
exit
area 0.0.0.1
no loopfree-alternate-exclude
blackhole-aggregate
stub
summaries
default-metric 1
exit
interface "Int-125668019-1696182375"
no passive
interface-type broadcast
advertise-subnet
priority 1
hello-interval 10
dead-interval 40
retransmit-interval 5
transit-delay 1
no mtu
no metric
no authentication-type
no authentication-key
no bfd-enable
no loopfree-alternate-exclude
no shutdown
exit
exit

6.2. OSPF 391

 VNS User Guide, Release 5.3.2

no shutdown

###################################

vsc# tools vswitch 10.15.1.254 command "nuage-ospf-show status 125668019"

===============================================================================
OSPF Status
===============================================================================
OSPF Cfg Router Id : 64.152.193.19
OSPF Oper Router Id : 64.152.193.19
OSPF Version : 2
OSPF Admin Status : Enabled
OSPF Oper Status : Enabled
Graceful Restart : Disabled
GR Helper Mode : Disabled
Preference : 10
External Preference : 150
Backbone Router : True
Area Border Router : True
AS Border Router : True
Opaque LSA Support : True
Traffic Engineering Support : False
RFC 1583 Compatible : True
Demand Exts Support : False
In Overload State : False
In External Overflow State : False
Exit Overflow Interval : 0
Last Overflow Entered : Never
Last Overflow Exit : Never
External LSA Limit : -1
Reference Bandwidth : 100,000,000 Kbps
Init SPF Delay : 1000 msec
Sec SPF Delay : 1000 msec
Max SPF Delay : 10000 msec
Min LS Arrival Interval : 1000 msec
Init LSA Gen Delay : 5000 msec
Sec LSA Gen Delay : 5000 msec
Max LSA Gen Delay : 5000 msec
Last Ext SPF Run : Never
Ext LSA Cksum Sum : 0x0
OSPF Last Enabled : 0x0
Unicast Import : True
Multicast Import : False
Export Policies : None
Import Policies : None
OSPF Ldp Sync Admin Status : Enabled
LDP-over-RSVP : Disabled
RSVP-Shortcut : Disabled
Advertise-Tunnel-Link : Disabled
LFA : Disabled
Export Limit : 0
Export Limit Log Percent : 0
Total Exp Routes : 0
Ignore DN Bit : False
Suppress DN Bit : False
VPN Tag : 0

6.2. OSPF 392

 VNS User Guide, Release 5.3.2

===============================================================================

###################################

vsc# tools vswitch 10.15.1.254 command "nuage-ospf-show neighbor 125668019"

===============================================================================
OSPF Neighbors
===============================================================================
Interface-Name Rtr Id State Pri RetxQ TTL
Area-Id
-------------------------------------------------------------------------------
Int-125668019-602706402 20.0.2.100 Full 1 0 32
0.0.0.0
-------------------------------------------------------------------------------
No. of Neighbors: 1
===============================================================================

####################################

vsc# tools vswitch 10.15.1.254 command "nuage-ospf-show neighbor-detail 125668019"

===============================================================================
OSPF Neighbors
===============================================================================
-------------------------------------------------------------------------------
Neighbor Rtr Id : 20.0.2.100 Interface: Int-125668019-602706402
-------------------------------------------------------------------------------
Neighbor IP Addr : 20.0.2.100
Local IF IP Addr : 20.0.2.1
Area Id : 0.0.0.0
Designated Rtr : 64.152.193.19 Backup Desig Rtr : 20.0.2.100
Neighbor State : Full Priority : 1
Retrans Q Length : 0 Options : - E - - - - O --
Events : 4 Last Event Time : - E - - - - O --
Up Time : 0d 00:04:30 Time Before Dead : 34 sec
GR Helper : Not Helping GR Helper Age : 0 sec
GR Exit Reason : None GR Restart Reason: Unknown
Bad Nbr States : 1 LSA Inst fails : 0
Bad Seq Nums : 0 Bad MTUs : 0
Bad Packets : 0 LSA not in LSDB : 0
Option Mismatches: 0 Nbr Duplicates : 0
Num Restarts : 0 Last Restart at : Never
===============================================================================

####################################

vsc# tools vswitch 10.15.1.254 command "nuage-ospf-show routes 125668019"

================================================================================
OSPFv2 Routing Table
================================================================================
Destination Type(Dest) Stat
NHIP NHIF Cost[E2]
--------------------------------------------------------------------------------
20.0.0.0/24 IA (STUB) D (F)
DIRECT 2 10000
20.0.1.0/24 IA (STUB) D (F)

6.2. OSPF 393

 VNS User Guide, Release 5.3.2

DIRECT 3 10000
20.0.2.0/24 IA (NET) D (F)
DIRECT 4 10000
20.0.3.0/24 IA (STUB) D (F)
DIRECT 5 10000

20.0.2.100/0 IA (AB-AS) N (N)

20.0.2.100 4 10000

--------------------------------------------------------------------------------
5 OSPFv2 routes found (5 paths)
Stat: D = direct N = not direct
(RTM stat):(R) = added (F) = add failed
(N) = not added (D) = policy discarded
================================================================================

####################################

vsc# tools vswitch 10.15.1.254 command "nuage-ospf-show routes-detail 125668019"

================================================================================
OSPFv2 Routing Table (detailed)
================================================================================
Destination Type(Dest) Stat
NHIP NHIF Cost[E2] Area Tunnel-Information
--------------------------------------------------------------------------------
20.0.0.0/24 IA (STUB) D (F)
DIRECT 2 10000 0.0.0.0
20.0.1.0/24 IA (STUB) D (F)
DIRECT 3 10000 0.0.0.1
20.0.2.0/24 IA (NET) D (F)
DIRECT 4 10000 0.0.0.0
20.0.3.0/24 IA (STUB) D (F)
DIRECT 5 10000 0.0.0.1

20.0.2.100/0 IA (AB-AS) N (N)

20.0.2.100 4 10000 0.0.0.0

--------------------------------------------------------------------------------
5 OSPFv2 routes found (5 paths)
Stat: D = direct N = not direct
(RTM stat):(R) = added (F) = add failed
(N) = not added (D) = policy discarded
================================================================================

####################################

vsc# tools vswitch 10.15.1.254 command "nuage-ospf-show area 125668019"

===============================================================================
OSPF Areas (Detailed)
===============================================================================
-------------------------------------------------------------------------------
Area Id: 0.0.0.0
-------------------------------------------------------------------------------
Area Id : 0.0.0.0 Type : Standard
Key Rollover Int.: 10 LFA : Include

6.2. OSPF 394

 VNS User Guide, Release 5.3.2

Virtual Links : 0 Total Nbrs : 1

Active IFs : 2 Total IFs : 2
Area Bdr Rtrs : 2 AS Bdr Rtrs : 2
SPF Runs : 11 Last SPF Run : 2
Router LSAs : 2 Network LSAs : 1
Summary LSAs : 2 Asbr-summ LSAs : 0
Nssa ext LSAs : 0 Area opaque LSAs : 0
Total LSAs : 5 LSA Cksum Sum : 0x18094
Blackhole Range : True Unknown LSAs : 0
-------------------------------------------------------------------------------
Area Id: 0.0.0.1
-------------------------------------------------------------------------------
Area Id : 0.0.0.1 Type : Stub
Default Cost : 1 Import Summary : Send Summary
Key Rollover Int.: 10 LFA : Include
Virtual Links : 0 Total Nbrs : 0
Active IFs : 2 Total IFs : 2
Area Bdr Rtrs : 1 AS Bdr Rtrs : 0
SPF Runs : 10 Last SPF Run : 0
Router LSAs : 1 Network LSAs : 0
Summary LSAs : 3 Asbr-summ LSAs : 0
Nssa ext LSAs : 0 Area opaque LSAs : 0
Total LSAs : 4 LSA Cksum Sum : 0x1e47e
Blackhole Range : True Unknown LSAs : 0
===============================================================================

####################################

vsc# tools vswitch 10.15.1.254 command "nuage-ospf-show interface 125668019"

===============================================================================
OSPF Interfaces (Detailed)
===============================================================================
-------------------------------------------------------------------------------
Interface : Int-125668019-1657287575
-------------------------------------------------------------------------------
IP Address : 20.0.0.1
Area Id : 0.0.0.0 Priority : 1
Hello Intrvl : 10 sec Rtr Dead Intrvl : 40 sec
Retrans Intrvl : 5 sec Poll Intrvl : 120 sec
Cfg Metric : 0 Advert Subnet : True
Transit Delay : 1 Auth Type : None
Passive : False Cfg MTU : 0
LFA : Include
IPsec InStatSA : IPsec OutStatSA :
IPsec InStatSATmp:
Admin Status : Enabled Oper State : Designated Rtr
Designated Rtr : 64.152.193.19 Backup Desig Rtr : 0.0.0.0
IF Type : Broadcast Network Type : Stub
Oper MTU : 1500 Last Enabled : Stub
Oper Metric : 10000 Bfd Enabled : No
Te Metric : 10000 Te State : Down
Admin Groups : None
Ldp Sync : outOfService Ldp Sync Wait : Disabled
Ldp Timer State : Disabled Ldp Tm Left : 0
Nbr Count : 0 If Events : 2
Tot Rx Packets : 0 Tot Tx Packets : 3436
Rx Hellos : 0 Tx Hellos : 3436

6.2. OSPF 395

 VNS User Guide, Release 5.3.2

Rx DBDs : 0 Tx DBDs : 0
Rx LSRs : 0 Tx LSRs : 0
Rx LSUs : 0 Tx LSUs : 0
Rx LS Acks : 0 Tx LS Acks : 0
Retransmits : 0 Discards : 0
Bad Networks : 0 Bad Virt Links : 0
Bad Areas : 0 Bad Dest Addrs : 0
Bad Auth Types : 0 Auth Failures : 0
Bad Neighbors : 0 Bad Pkt Types : 0
Bad Lengths : 0 Bad Hello Int. : 0
Bad Dead Int. : 0 Bad Options : 0
Bad Versions : 0 Bad Checksums : 0
LSA Count : 0 LSA Checksum : 0x0

-------------------------------------------------------------------------------
Interface : Int-125668019-1696182375
-------------------------------------------------------------------------------
IP Address : 20.0.1.1
Area Id : 0.0.0.1 Priority : 1
Hello Intrvl : 10 sec Rtr Dead Intrvl : 40 sec
Retrans Intrvl : 5 sec Poll Intrvl : 120 sec
Cfg Metric : 0 Advert Subnet : True
Transit Delay : 1 Auth Type : None
Passive : False Cfg MTU : 0
LFA : Include
IPsec InStatSA : IPsec OutStatSA :
IPsec InStatSATmp:
Admin Status : Enabled Oper State : Designated Rtr
Designated Rtr : 64.152.193.19 Backup Desig Rtr : 0.0.0.0
IF Type : Broadcast Network Type : Stub
Oper MTU : 1500 Last Enabled : Stub
Oper Metric : 10000 Bfd Enabled : No
Te Metric : 10000 Te State : Down
Admin Groups : None
Ldp Sync : outOfService Ldp Sync Wait : Disabled
Ldp Timer State : Disabled Ldp Tm Left : 0
Nbr Count : 0 If Events : 2
Tot Rx Packets : 0 Tot Tx Packets : 3436
Rx Hellos : 0 Tx Hellos : 3436
Rx DBDs : 0 Tx DBDs : 0
Rx LSRs : 0 Tx LSRs : 0
Rx LSUs : 0 Tx LSUs : 0
Rx LS Acks : 0 Tx LS Acks : 0
Retransmits : 0 Discards : 0
Bad Networks : 0 Bad Virt Links : 0
Bad Areas : 0 Bad Dest Addrs : 0
Bad Auth Types : 0 Auth Failures : 0
Bad Neighbors : 0 Bad Pkt Types : 0
Bad Lengths : 0 Bad Hello Int. : 0
Bad Dead Int. : 0 Bad Options : 0
Bad Versions : 0 Bad Checksums : 0
LSA Count : 0 LSA Checksum : 0x0

-------------------------------------------------------------------------------
Interface : Int-125668019-602706402
-------------------------------------------------------------------------------

6.2. OSPF 396

 VNS User Guide, Release 5.3.2

IP Address : 20.0.2.1
Area Id : 0.0.0.0 Priority : 1
Hello Intrvl : 10 sec Rtr Dead Intrvl : 40 sec
Retrans Intrvl : 5 sec Poll Intrvl : 120 sec
Cfg Metric : 0 Advert Subnet : True
Transit Delay : 1 Auth Type : None
Passive : False Cfg MTU : 0
LFA : Include
IPsec InStatSA : IPsec OutStatSA :
IPsec InStatSATmp:
Admin Status : Enabled Oper State : Designated Rtr
Designated Rtr : 64.152.193.19 Backup Desig Rtr : 20.0.2.100
IF Type : Broadcast Network Type : Transit
Oper MTU : 1500 Last Enabled : Transit
Oper Metric : 10000 Bfd Enabled : No
Te Metric : 10000 Te State : Down
Admin Groups : None
Ldp Sync : outOfService Ldp Sync Wait : Disabled
Ldp Timer State : Disabled Ldp Tm Left : 0
Nbr Count : 1 If Events : 2
Tot Rx Packets : 62 Tot Tx Packets : 3450
Rx Hellos : 51 Tx Hellos : 3438
Rx DBDs : 3 Tx DBDs : 3
Rx LSRs : 1 Tx LSRs : 1
Rx LSUs : 3 Tx LSUs : 6
Rx LS Acks : 4 Tx LS Acks : 2
Retransmits : 0 Discards : 0
Bad Networks : 0 Bad Virt Links : 0
Bad Areas : 0 Bad Dest Addrs : 0
Bad Auth Types : 0 Auth Failures : 0
Bad Neighbors : 0 Bad Pkt Types : 0
Bad Lengths : 0 Bad Hello Int. : 0
Bad Dead Int. : 0 Bad Options : 0
Bad Versions : 0 Bad Checksums : 0
LSA Count : 0 LSA Checksum : 0x0

-------------------------------------------------------------------------------
Interface : Int-125668019-1458500558
-------------------------------------------------------------------------------
IP Address : 20.0.3.1
Area Id : 0.0.0.1 Priority : 1
Hello Intrvl : 10 sec Rtr Dead Intrvl : 40 sec
Retrans Intrvl : 5 sec Poll Intrvl : 120 sec
Cfg Metric : 0 Advert Subnet : True
Transit Delay : 1 Auth Type : None
Passive : False Cfg MTU : 0
LFA : Include
IPsec InStatSA : IPsec OutStatSA :
IPsec InStatSATmp:
Admin Status : Enabled Oper State : Designated Rtr
Designated Rtr : 64.152.193.19 Backup Desig Rtr : 0.0.0.0
IF Type : Broadcast Network Type : Stub
Oper MTU : 1500 Last Enabled : Stub
Oper Metric : 10000 Bfd Enabled : No
Te Metric : 10000 Te State : Down
Admin Groups : None
Ldp Sync : outOfService Ldp Sync Wait : Disabled

6.2. OSPF 397

 VNS User Guide, Release 5.3.2

Ldp Timer State : Disabled Ldp Tm Left : 0

Nbr Count : 0 If Events : 2
Tot Rx Packets : 0 Tot Tx Packets : 35
Rx Hellos : 0 Tx Hellos : 35
Rx DBDs : 0 Tx DBDs : 0
Rx LSRs : 0 Tx LSRs : 0
Rx LSUs : 0 Tx LSUs : 0
Rx LS Acks : 0 Tx LS Acks : 0
Retransmits : 0 Discards : 0
Bad Networks : 0 Bad Virt Links : 0
Bad Areas : 0 Bad Dest Addrs : 0
Bad Auth Types : 0 Auth Failures : 0
Bad Neighbors : 0 Bad Pkt Types : 0
Bad Lengths : 0 Bad Hello Int. : 0
Bad Dead Int. : 0 Bad Options : 0
Bad Versions : 0 Bad Checksums : 0
LSA Count : 0 LSA Checksum : 0x0
===============================================================================

####################################

vsc# tools vswitch 10.15.1.254 command "nuage-ospf-show database 125668019"

===============================================================================
OSPF Link State Database (Type : All)
===============================================================================
Type Area Id Link State Id Adv Rtr Id Age Sequence Cksum
-------------------------------------------------------------------------------
Router 0.0.0.0 20.0.2.100 20.0.2.100 534 0x80000008 0x4394
Router 0.0.0.0 64.152.193.19 64.152.193.19 383 0x8000001a 0x24f7
Network 0.0.0.0 20.0.2.1 64.152.193.19 534 0x80000001 0x204d
Summary 0.0.0.0 20.0.1.0 64.152.193.19 59 0x80000012 0x74dd
Summary 0.0.0.0 20.0.3.0 64.152.193.19 382 0x80000001 0x80e0
Router 0.0.0.1 64.152.193.19 64.152.193.19 352 0x8000001a 0x45ef
Summary 0.0.0.1 0.0.0.0 64.152.193.19 352 0x8000002b 0x790d
Summary 0.0.0.1 20.0.0.0 64.152.193.19 1503 0x80000011 0x9fb6
Summary 0.0.0.1 20.0.2.0 64.152.193.19 533 0x80000013 0x85cc
-------------------------------------------------------------------------------
No. of LSAs: 9
===============================================================================

####################################

vsc# tools vswitch 10.15.1.254 command "nuage-ospf-show database-detail 125668019"

===============================================================================
OSPF Link State Database (Type : All) (Detailed)
===============================================================================
-------------------------------------------------------------------------------
Router LSA for Area 0.0.0.0
-------------------------------------------------------------------------------
Area Id : 0.0.0.0 Adv Router Id : 20.0.2.100
Link State Id : 20.0.2.100 (335544932)
LSA Type : Router
Sequence No : 0x80000008 Checksum : 0x4394
Age : 593 Length : 36
Options : E

6.2. OSPF 398

 VNS User Guide, Release 5.3.2

Flags : ABR ASBR Link Count : 1

Link Type (1) : Transit Network
DR Rtr Id (1) : 20.0.2.1 I/F Address (1) : 20.0.2.100
No of TOS (1) : 0 Metric-0 (1) : 1000
-------------------------------------------------------------------------------
Router LSA for Area 0.0.0.0
-------------------------------------------------------------------------------
Area Id : 0.0.0.0 Adv Router Id : 64.152.193.19
Link State Id : 64.152.193.19 (1083752723)
LSA Type : Router
Sequence No : 0x8000001a Checksum : 0x24f7
Age : 441 Length : 48
Options : E
Flags : ABR ASBR Link Count : 2
Link Type (1) : Stub Network
Network (1) : 20.0.0.0 Mask (1) : 255.255.255.0
No of TOS (1) : 0 Metric-0 (1) : 10000
Link Type (2) : Transit Network
DR Rtr Id (2) : 20.0.2.1 I/F Address (2) : 20.0.2.1
No of TOS (2) : 0 Metric-0 (2) : 10000
-------------------------------------------------------------------------------
Network LSA for Area 0.0.0.0
-------------------------------------------------------------------------------
Area Id : 0.0.0.0 Adv Router Id : 64.152.193.19
Link State Id : 20.0.2.1 (335544833)
LSA Type : Network
Sequence No : 0x80000001 Checksum : 0x204d
Age : 593 Length : 32
Options : E
Network Mask : 255.255.255.0 No of Adj Rtrs : 2
Router Id (1) : 64.152.193.19 Router Id (2) : 20.0.2.100
-------------------------------------------------------------------------------
Summary LSA for Area 0.0.0.0
-------------------------------------------------------------------------------
Area Id : 0.0.0.0 Adv Router Id : 64.152.193.19
Link State Id : 20.0.1.0 (335544576)
LSA Type : Summary
Sequence No : 0x80000012 Checksum : 0x74dd
Age : 118 Length : 28
Options : E
Network Mask : 255.255.255.0 Metric-0 : 10000
-------------------------------------------------------------------------------
Summary LSA for Area 0.0.0.0
-------------------------------------------------------------------------------
Area Id : 0.0.0.0 Adv Router Id : 64.152.193.19
Link State Id : 20.0.3.0 (335545088)
LSA Type : Summary
Sequence No : 0x80000001 Checksum : 0x80e0
Age : 440 Length : 28
Options : E
Network Mask : 255.255.255.0 Metric-0 : 10000
-------------------------------------------------------------------------------
Router LSA for Area 0.0.0.1
-------------------------------------------------------------------------------
Area Id : 0.0.0.1 Adv Router Id : 64.152.193.19
Link State Id : 64.152.193.19 (1083752723)
LSA Type : Router
Sequence No : 0x8000001a Checksum : 0x45ef

6.2. OSPF 399

 VNS User Guide, Release 5.3.2

Age : 410 Length : 48

Options : None
Flags : ABR Link Count : 2
Link Type (1) : Stub Network
Network (1) : 20.0.1.0 Mask (1) : 255.255.255.0
No of TOS (1) : 0 Metric-0 (1) : 10000
Link Type (2) : Stub Network
Network (2) : 20.0.3.0 Mask (2) : 255.255.255.0
No of TOS (2) : 0 Metric-0 (2) : 10000
-------------------------------------------------------------------------------
Summary LSA for Area 0.0.0.1
-------------------------------------------------------------------------------
Area Id : 0.0.0.1 Adv Router Id : 64.152.193.19
Link State Id : 0.0.0.0 (0)
LSA Type : Summary
Sequence No : 0x8000002b Checksum : 0x790d
Age : 410 Length : 28
Options : None
Network Mask : 0.0.0.0 Metric-0 : 1
-------------------------------------------------------------------------------
Summary LSA for Area 0.0.0.1
-------------------------------------------------------------------------------
Area Id : 0.0.0.1 Adv Router Id : 64.152.193.19
Link State Id : 20.0.0.0 (335544320)
LSA Type : Summary
Sequence No : 0x80000011 Checksum : 0x9fb6
Age : 1561 Length : 28
Options : None
Network Mask : 255.255.255.0 Metric-0 : 10000
-------------------------------------------------------------------------------
Summary LSA for Area 0.0.0.1
-------------------------------------------------------------------------------
Area Id : 0.0.0.1 Adv Router Id : 64.152.193.19
Link State Id : 20.0.2.0 (335544832)
LSA Type : Summary
Sequence No : 0x80000013 Checksum : 0x85cc
Age : 592 Length : 28
Options : None
Network Mask : 255.255.255.0 Metric-0 : 10000
===============================================================================

####################################

vsc# tools vswitch 10.15.1.254 command "nuage-ospf-show stats 125668019"

===============================================================================
OSPF Statistics
===============================================================================
Rx Packets : 81 Tx Packets : 10430
Rx Hellos : 69 Tx Hellos : 10417
Rx DBDs : 3 Tx DBDs : 3
Rx LSRs : 1 Tx LSRs : 1
Rx LSUs : 3 Tx LSUs : 7
Rx LS Acks : 5 Tx LS Acks : 2
New LSAs Recvd : 0 New LSAs Orig : 10
Ext LSAs Count : 0 No of Areas : 2
No of Interfaces : 4 No of Neighbors : 1
Retransmits : 0 Discards : 0

6.2. OSPF 400

 VNS User Guide, Release 5.3.2

Bad Networks : 0 Bad Virt Links : 0

Bad Areas : 0 Bad Dest Addrs : 0
Bad Auth Types : 0 Auth Failures : 0
Bad Neighbors : 0 Bad Pkt Types : 0
Bad Lengths : 0 Bad Hello Int. : 0
Bad Dead Int. : 0 Bad Options : 0
Bad Versions : 0 Bad Checksums : 0
Failed SPF Attempts: 0 Bad MTUs : 0
CSPF Requests : 0 CSPF Request Drops : 0
CSPF Path Found : 0 CSPF Path Not Found: 0
Total SPF Runs : 11 Total LFA SPF Runs : 0
===============================================================================
####################################

vsc# nuage-router-show rtm-routes 125668019

===============================================================================
Route Table (Router: Vrf 2)
===============================================================================
Dest Prefix[Flags] Type Proto Age Pref
Next Hop[Interface Name] Metric
-------------------------------------------------------------------------------
0.0.0.0/0 Remote Static 09h00m29s 255
0
20.0.0.0/24 Local NVC 00h11m43s 0
0
20.0.0.104/32 Local NVC 00h11m43s 0
0
20.0.1.0/24 Local NVC 00h11m43s 0
0
20.0.2.0/24 Local NVC 00h11m42s 0
0
20.0.2.100/32 Local NVC 00h11m33s 0
0
20.0.3.0/24 Local NVC 00h11m42s 0
0
20.0.4.0/24 Remote BGP VPN 00h10m48s 170
Local VRF 0
20.0.5.0/24 Remote BGP VPN 00h10m48s 170
Local VRF 0
20.0.6.0/24 Remote BGP VPN 00h10m48s 170
Local VRF 0
20.0.7.0/24 Remote BGP VPN 00h10m48s 170
Local VRF 0
20.0.8.0/24 Remote BGP VPN 00h10m48s 170
Local VRF 0
-------------------------------------------------------------------------------
No. of Routes: 12
Flags: L = LFA nexthop available B = BGP backup route available
n = Number of times nexthop is repeated * = Virtual Nexthop
===============================================================================

6.2. OSPF 401

 CHAPTER

SEVEN

ENCRYPTION

• Group-Key IPsec (page 403)

• IKE IPsec (page 421)

402
 VNS User Guide, Release 5.3.2

7.1 Group-Key IPsec

• Overview (page 403)

• Terminology (page 404)
• Use Case: VXLANoIPsec (page 405)
• IPsec Architecture (page 406)
• Overview of Cryptographic Process (page 407)
– SEK Processing (page 407)
– Seed Processing (page 407)
– TEK Processing (page 408)
– Processing of Forced Rekey and Revoke Events (page 408)
– Processing for Multi-tenant NSG-BR (page 409)
– Processing during Controllerless Mode (page 409)
• Workflows (page 409)
– High-Level Feature Workflow (page 409)
– Detailed Provisioning Workflow (page 410)

* Enter System Configuration (page 410)

* Enable Security for Organization (page 411)
* KS Configuration (page 412)
* Create Secure L3 Domain/Zone/Subnet (page 413)
* Create Secure L2 Domain (page 415)
• VxLANoIPsec QoS Marking (page 416)
– VxLAN QoS Marking (page 416)

* Network QoS Configuration on VSC (page 418)

• Interworking with NAT-T (page 419)
• CLI Command Sample Output (page 419)

7.1.1 Overview

Customers require encryption for traffic that traverses over WAN networks between branch sites or between branch
and headquarters. Nuage VNS incorporates a unique solution that provides IPsec encryption for WAN traffic over both
public and private networks.
VNS IPsec enables organizations to meet security compliance requirements where encryption is a mandatory require-
ment. Supported topologies include a highly scalable architecture for fully meshed branch to branch and hub-and-
spoke secure communication.
The feature enables:

7.1. Group-Key IPsec 403

 VNS User Guide, Release 5.3.2

• Traffic in a domain (L3-domain/unmanaged L2-domain) to be encrypted, which will effectively encrypt traffic
to/from all NSGs participating in a single domain.
• Traffic destined to a particular subnet to be encrypted.
• Traffic to/from multi-tenanted NSG-BR to other NSGs is encrypted using tenant specific keys that the destination
NSG belongs to.
From a VNS perspective, encryption is provided as a transport function that offers an encrypted communication path
between two or more end-points upon which overlay services (L3-domains/L2-domains) can be transparently enacted.
Thus, the VNS service layer behaviour is consistent regardless of whether encryption is enabled or not.
Feature highlights:
• Encryption is an addition to the existing VSP service policy framework with new constructs.
• Encryption policy is simple to configure and enforce. Complex IPsec parameters can be provisioned with a click
of a button or RESTful API and enforcement is distributed to all the VNS endpoints in a scalable way. Design
allows for very granular policy rules for encryption (subnet versus vPort)
• Interworking with NAT in tunnel-less mode (i.e no IKE)
• Designed to support both managed and unmanaged Key Server (KS) model of operations.
• All control/management communications to and from NSG is authenticated and encrypted.
• All data-plane communications between NSGs is encrypted (no local offload) regardless of transport type (pri-
vate or Internet).
• Support for sequence-based anti-replay in a group key architecture which is more secure than time-based.
• Perfect Forward Secrecy whereby current seed cannot be used to determine subsequent encryption keys or
subsequent seed.
• Revocation for a rogue NSG is built into the architecture.
• NAT traversal for ESP packets is provided as an inherent functionality without the need for IKE. See IPsec with
NAT-T (page 419)).
• Scalable implementation:
– Control plane designed to minimize impact on CPU for data plane functions
– Hardware acceleration for data plane
– Control plane separate from data plane. Thus, data path encryption is not disrupted even if connectivity to
KS is lost.
– Expensive computations such as key generation off-loaded to endpoint NSGs. Thus, the KS is lightweight
and a rekey for a large NSG network does not cripple the KS.
The VNS IPsec capability assumes that all encrypting/decrypting endpoints are VNS NSGs (physical or virtual form
factors). Key generation and distribution is based on a Nuage proprietary mechanism and Internet Key Exchange
(IKE) is not used for negotiating the encryption attributes between endpoints.For interoperability with 3rd party IPsec
gateways, refer to IKE-based IPsec (page 421) chapter.

7.1.2 Terminology

• NSG-P: Network Services Gateway appliance (physical) typically deployed at a branch site
• NSG-V: Network Services Gateway appliance (virtual) typically deployed at a CO or in a DC environment
• Key Server (KS): Entity that is responsible for generating encryption keys

7.1. Group-Key IPsec 404

 VNS User Guide, Release 5.3.2

• Tunnel-based IPsec : ESP based implementation that relies on IKE for authentication, key generation and dis-
tribution.
• Group-key based IPsec: ESP based implementation that relies on KS entity to handle end-point authentication
(member registration), key (re)generation and distribution.
• Security Association (SA): A policy that governs encryption and encapsulation parameters between a set of
IPsec peers belonging to a specific domain. Multiple domains on a single NSG will have different SAs.
• Seed Encryption Key (SEK): A symmetric key that is used to encrypt the seed material during transport to the
NSG via the VSD and VSC. The public key of the NSG, which is generated as part of the bootstrap process, is
used by the KS to encrypt the SEK. The encrypted SEK is distributed to every NSG. The NSG uses the SEK to
decrypt the seed material.
• Seed: Seed material that is generated by the KS, transported to NSG with additional attributes. Seed is encrypted
using the SEK and stored on the KS. The seed material is used by NSG to generate the Traffic Encryption Key
(TEK).
• Traffic Encryption Key (TEK) or Enterprise Group Key: Symmetric encryption keys that are used to encrypt
data traffic sent between NSGs

7.1.3 Use Case: VXLANoIPsec

The deployment model supports VxLANoIPsec tunnels between NSG end-points as a full-mesh or hub-n-spoke:
• Full Mesh: Topology is referred to as full-mesh when traffic is encrypted between NSG end-points without
the use of point-to-point tunnels. In order to accomplish the tunnel-less communication, keying material is
distributed to the NSGs using a Nuage proprietary method.

Fig. 7.1: Full Mesh Topology

• Hub-n-Spoke: Is a variation of the full mesh topology when encrypted traffic flow is between a hub site and
branch sites. In other words there is no branch to branch communication.

7.1. Group-Key IPsec 405

 VNS User Guide, Release 5.3.2

Fig. 7.2: Hub-n-Spoke Topology

7.1.4 IPsec Architecture

For IPsec, a new functional component is added to the existing VNS architecture. The component is a centralized Key
Server (KS) that is responsible for enforcing the encryption policy and for key management.

Fig. 7.3: IPsec Architecture

Under the current VNS deployment model, the KS is assumed to be a trusted entity of the VSD and is deployed as
a module within VSD. It is referred to as a managed KS whereby the enterprise administrator manages the security
policies for the organization, as well as the encryption policies on the KS. A KS that is a separate entity from the VSD
and is untrusted is referred to as an unmanaged KS and will be supported in a future release.
The primary functions of the KS are:
• Generation of encryption keys
• Generation of Seed
• Lifecycle management of keys/seed
• Enterprise/NSG registration, authentication and authorization
• X.509 certificate awareness
In the case of a managed KS, the setup and startup/initialization process occurs as part of the VSD startup. The
KS authenticates to the VSD using X.509 certificate generated by the VSD as root CA. Upon completion of the
initialization of VSD, the KS is in an operationally ‘ready’ state to generate the keying material for the enterprise. KS
subscribes to the event notification bus using the same key to listen to rekey and NSG revocation events which triggers
key/seed re-generation.

7.1. Group-Key IPsec 406

 VNS User Guide, Release 5.3.2

The IPsec capability is selected at the Organization level via a security option. The KS processes startup when the
security option is enabled. Once the KS process is started, the KS periodically generates for the Organization:
• SEK which is an object consisting of an internally generated key ID and configured encryption attributes. The
public key of the NSG, which is generated as part of the bootstrap process, is used by the KS to encrypt the
SEK. The SEK is stored on the VSD for delivery to NSGs.
• Seed which is the keying material for generating traffic encryption keys on the NSG. The seed material is
encrypted (using the SEK) and signed by the KS and stored on the VSD for delivery to NSGs.

Note: The generation of the SEK and the Seed occurs even if there are no NSGs present in an active state for that
enterprise.

Security policies that are configured in the VSD are pushed to the NSGs as follows:
• If policies are present at bootstrapping time, then they are sent as part of the infrastructure configuration once
the NSGs are authenticated. Keys are pushed using vPort resolution procedures.
• If policies are configured post-bootstrapping, they are pushed using same procedures used to update infrastruc-
ture configuration. Keys are pushed using domain/subnet configuration update procedures.
Key-distribution to the NSGs is handled via VSC unicasting over OF-TLS channel to individual NSGs that are part of
the same member group.

7.1.5 Overview of Cryptographic Process

The KS generates the cryptographic data, namely the SEK, Seed and associated keying material, which is then dis-
tributed securely to all the active NSGs by the VSD. The NSG generates TEK from the Seed, and the TEK is used to
encrypt the data path.

SEK Processing

The SEK is used to encrypt/decrypt the Seed. For secure transport to the NSG, the SEK and related attributes are
encrypted and signed by the KS. The encrypted SEK is transported to the NSGs over XMPP-TLS (VSD -> VSC)
and OF-TLS (VSC -> NSG). The cryptographic parameters for the SEK are: (see SEK parameters (page 413) for
configuration details):
• SEK Payload Encryption Algorithm: RSA (Asymmetric encryption)
• SEK Payload Encryption Key: NSG’s Public Key (received as part of the bootstrapping process)
• SEK Payload Signature Algorithm: Configured by CSPRoot as part of the System Configuration. Cannot be
changed by Enterprise admin. The KS uses its Private Key for signing.
Upon receipt of the encrypted and signed SEK, the NSG processes the received data as follows:
• Verifies the signature using the KS Public Key that is received as part of the bootstrapping process.
• If signature is valid, then the NSG uses its private key to decrypt and retrieve the SEK data which it then uses
for Seed processing.

Seed Processing

Secure transport of Seed material to NSG utilizes encryption, authentication and digital signing. Thus, the crypto-
graphic parameters for the Seed are (see Seed parameters (page 413) for configuration details):

7.1. Group-Key IPsec 407

 VNS User Guide, Release 5.3.2

• Seed Payload Encryption Algorithm: Default value is configured by CSPRoot, but can be changed by Enterprise
Admin to another symmetric algorithm option.
• Seed Payload Encryption Key: SEK (generated by KS)
• Seed Payload Authentication Algorithm: Default value is configured by CSPRoot, but can be changed by En-
terprise Admin to another authentication algorithm option.
• Seed Payload Authentication Key: Generated by KS as part of the SEK data
• Seed Payload Signature Algorithm: Default value is configured by CSPRoot, but can be changed by Enterprise
Admin to another asymmetric algorithm option. The KS uses its Private Key for signing.
When the encrypted and signed Seed material is received by the NSG, it is processed as follows:
• Verify the authenticity of the received digest using the authentication algorithm and authentication key
received as part of SEK data.
• If authenticity is verified, then verify the signature using the KS Public Key.
• If signature is valid, then use SEK (retrieved as part of SEK processing) to retrieve the Seed material.

TEK Processing

TEK generation and processing is on the NSGs. All NSGs generate the same TEK from the Seed using a Nuage pro-
prietary implementation. The TEK is used to encrypt the data traffic sent between the NSGs. The TEK cryptographic
parameters are (see TEK Parameters (page 413) for configuration details):
• TEK Payload Encryption Algorithm: Default value is configured by CSPRoot, but can be changed by Enterprise
Admin to another symmetric algorithm option.
• TEK Payload Encryption Key: Generated on the NSG using the Seed and some additional Seed material.
• TEK Payload Authentication Algorithm: Default value is configured by CSPRoot, but can be changed by En-
terprise Admin to another authentication algorithm option.
• TEK Payload Authentication Key: The key for HMAC is generated by NSG as part of the TEK data.

Processing of Forced Rekey and Revoke Events

Under certain scenarios, an Enterprise admin may want to refresh all the cryptographic keys in the network. The
capability is supported using a Force Re-Key function that can be invoked as follows:
1. In the Enterprise view, naviagate to Dashboard -> Key Server Monitor. On the far right towards the top is Force
Re-key button.

Fig. 7.4: Force Re-Key across Enterprise

7.1. Group-Key IPsec 408

 VNS User Guide, Release 5.3.2

In addition, when a NSG is revoked, it is treated as a security event, and the SEK and Seed are regenerated by the KS
and distributed to all existing NSGs for immediate use. No explicit user action is required.

Processing for Multi-tenant NSG-BR

In order to support encryption for a multi-tenant NSG-BR, the KS generates tenant (enterprise) specific SEK and Seed
data. All seed material lifecycle events (creation, rekey, deletion, refresh, revoke) are handled on a per enterprise basis.
On a multi-tenant NSG-BR, traffic to other NSGs is encrypted using the keys for the enterprise to which the destination
NSG belongs. IPsec policies and traffic keys are associated with an enterprise. In other words, NSGs beloning to
different enterprises will not share the same traffic keys.
Revocation of a multi-tenant NSG-BR will trigger a rekey on all NSGs which share the enterprises with the revoked
NSG-BR.

Processing during Controllerless Mode

In the event of a connectivity loss between NSG and VSC, or between VSC and VSD, the cryptographic data (Seed
etc.) may not be delivered to an NSG which results in keys not being available for encrypting traffic and hence
forwarding will stop.
Refer to IPsec Key Considerations (page 104) for decription of cryptographic processing using disaster recovery Seed
during controllerless contingency.

7.1.6 Workflows

High-Level Feature Workflow

The following is a high-level workflow of the cryptographic processing for enabling IPsec on NSG end-points (assum-
ing that the VSD is installed and the managed KS is also created and started).
• CSPRoot enters the default KS configuration parameters as part of the System Configuration (Enter System
Configuration (page 410)).
• Administrator enables the Organization for security (Enable Security for Organization (page 411)).
• Administrator configures the KS for the Organization by selecting cryptographic attributes related to SEK, Seed
and TEK management (KS Configuration (page 412)).
• Once KS configuration is complete, KS starts periodic generation of SEK and seed material.
• Administrator defines Domain/Zone/Subnet and enables security (Create secure L3 Domain/Zone/Subnet
(page 413)). Unmanaged L2 domains are also supported (Create secure L2 Domain (page 415)))
• All NSGs receives the SEK and encrypted seed material from the KS. Each NSG decrypts the seed using the
SEK, and generates the traffic encryption key from the seed. If a new NSG gets bootstrapped, it joins the domain
and receives the SEK and Seed from the KS.
• NSGs encrypt/decrypt the traffic data using the TEK (since all generate the same TEK for the shared Seed
material).

7.1. Group-Key IPsec 409

 VNS User Guide, Release 5.3.2

Detailed Provisioning Workflow

Enter System Configuration

Once the VSD is installed, the CSPRoot sets up the default values for the KS attributes through the system configu-
ration. The enterprise administrator has the choice of either keeping these default values or specifying new values via
KS Configuration (page 412).

1. As CSPRoot, open Data Center Configuration ( ). Under the Settings tab, click on icon which opens
the System Configuration page. The majority of the system configuration variables should not be modified
without discussion with Nuage Networks Customer Support. This section describes only the variables required
for IPsec related configuration. Refer to the current version of the Nuage VSP User Guide for the remaining
parameters on the System Configuration page.
2. Enter the values for the variables in the Key Server Default Values section. These will be the defaults for the
Organization’s KS settings. The values in the Figure below are example values. The recommended defaults are
provided along with the detailed parameter explanation below.

Fig. 7.5: System Configuration for IPsec

Enable Key Server Monitor is a check box to indicate whether key server monitoring should be enabled from the
operating system command line. This is useful to the CSPRoot for troubleshooting. Recommended default is Enable.
Seed Payload Encryption Algorithm: Is the symmetric encryption algorithm. Supported options are 3DES CBC,
AES-128 CBC, AES-256 CBC. Recommended default is AES-256 CBC.
Seed Payload Signature Algorithm: Is the signing algorithm. Supported options are SHA-1 with RSA, SHA-221
with RSA, SHA-256 with RSA, SHA-384 with RSA, SHA-512 with RSA. Recommended default is RSA SHA-1.
Seed Generation Interval: Is the time in seconds before the expiry of the seed that a new seed must be generated.
Minimum Seed Generation Interval: Minimum value that can be configured for seed generation interval.
Minimum Seed Lifetime: Minimum value that can be configured for the seed lifetime.
Seed Lifetime: Is the duration in seconds after which the seed expires and must be regenerated. Valid range is 60
minutes to 2 days. In the event of a forced rekey or revoke event, the Seed is refreshed before the expiry of the lifetime.

7.1. Group-Key IPsec 410

 VNS User Guide, Release 5.3.2

SEK Payload Signature Algorithm: Is the signing algorithm. Supported options are RSA SHA-1 / SHA-221 / SHA-
256 / SHA-384 / SHA-512 SEK Generation Interval: Is the time in seconds before the expiry of SEK lifetime
that a new SEK is generated. It is the overlap interval when the old and new SEKs will coexist. However, only one
will be active at any given time. The interval value depends on the SEK lifetime value and the number of NSGs.
Recommended default is 1200 seconds.
Minimum SEK Generation Interval: Minimum value that can be configured for SEK generation interval. The
CSPRoot sets this value and it establishes the lowest value that can be configured by the Enterprise administrator in
the KS configuration.
SEK Lifetime: Is the duration in seconds after which the SEK expires and must be refreshed. In the event of a forced
rekey or revoke event, the SEK is refreshed before the expiry of the lifetime.
Minimum SEK Lifetime: Minimum value that can be configured for SEK lifetime. Traffic Authentication Al-
gorithm: Is the IPsec transform hashing algorithm. Supported options are HMAC SHA-1 / SHA-256 / SHA-384 /
SHA-512.
Traffic Encryption Algorithm: Is the IPsec transform encryption algorithm. Supported options are 3DES CBC,
AES-128 CBC, AES-192 CBC, AES-256 CBC.
Traffic Encryption Key Lifetime: Is the duration in seconds after which the TEK expires and must be refreshed.
In the event of a forced rekey or revoke event, the TEK is refreshed before the expiry of the lifetime as soon as a
new Seed/SEK is received. The TEK is used for outbound encryptions as soon as it is generated and any security
associations based on the old TEK are deleted.

Note: TEK Lifetime <= Seed Lifetime <= SEK Lifetime.

Generation Interval On Forced ReKey: Is the duration in seconds after a forced rekey event that a newly generated
seed and SEK become active.
Generation Interval On Revoke: Is the duration in seconds after a NSG revoke event notification that a newly
generated seed and SEK become active.

Enable Security for Organization

The IPsec capability is selected at the Organization level via a security option in the Organization Profile referred to as
Encryption Management Mode. In order to enable/disable Encryption on an Organization it must be associated with
the appropriate Organization Profile that has the security option Enabled/Disabled using the following steps:

1. As CSPRoot, open Data Center Configuration ( ). Under the Organization Policies tab, click on for
Organization Profiles. To add a new profile, click the plus icon (+). The New Organization Profile popup
appears.
2. The option in the popup related to the IPsec feature is Management Mode. Default is Disabled. Select Enabled
to create a profile that can be associated with an Organization that requires encryption. All other configuration
parameters are explained in the current version of the VSP User Guide.

7.1. Group-Key IPsec 411

 VNS User Guide, Release 5.3.2

Fig. 7.6: New Organization Profile

3. The newly created Organization Profile can be associated with an existing or new Organization to enable security
encryption. This architecture allows a CSPRoot to create multiple infrastructure encryption profiles, each with
varying attributes, to address 3rd party gateway interoperability in the future when such a capability is supported.

KS Configuration

The KS is configured by specifying all the the crypto attributes required for SEK, Seed and TEK generation for that
organization.

1. In the Organization view, go to the Settings tab and click on icon in the left sidebar for Key Server Config-
uration. The Key Server Configuration popup appears. Note it is a scrollable panel in the VSD UI, but a split
view is shown here for readability.

7.1. Group-Key IPsec 412

 VNS User Guide, Release 5.3.2

Fig. 7.7: KS Configuration

2. Enter the information related to SEK (Generation, Authentication, Encryption and Signature), Seed Generation,
and TEK (Generation, Authentication, Encryption). Defaults selected by the CSPRoot as part of the System
Configuration will be displayed.
Refer to System Configuration (page 410) section for detailed definitions for the following: SEK Parameters
• SEK Generation Interval
• SEK Lifetime
• Seed Payload Authentication Algorithm
• Seed Payload Encryption Algorithm
• Seed Payload Signing Algorithm
Seed Parameters:
• Seed Generation Interval
• Seed Lifetime
Traffic Encryption Key Parameters
• Lifetime
• Authentication Algorithm
• Encryption Algorithm

Create Secure L3 Domain/Zone/Subnet

The Enterprise administrator defines Layer 3 networks using Domain templates that contains Zones and Subnets (refer
to current version of Nuage VSP User Guide for provisioning details). For an Organization that is enabled for security,
encryption can be specified at the Domain, Zone, and Subnet level. The rules for enabling encryption on the individual
Domain/Zone/Subnet objects are as follows:

7.1. Group-Key IPsec 413

 VNS User Guide, Release 5.3.2

• Each enterprise must have at least one Domain template defined by the administrator.
• Encryption can be enabled or disabled at the Domain template level. The Domain instance inherits the encryp-
tion settings from the Domain template and cannot be changed at the Domain instance level.
Enable encryption as follows:

1. In the Organization view, select Networks -> Layer 3 Domains , and select an L3 domain template.

2. In the center panel, select the template icon , and in the far right panel specify the encryption parameter
under the tab icon .

Fig. 7.8: Enabling Encryption for L3 Domain Template

• Encryption can be enabled, disabled, or inherited at the Zone, Subnet level. However, if there is a template
that defines Domain/Zone/Subnet along with encryption setting, then an instance created from such a template
will inherit all the encryption settings and cannot be changed. On the other hand if only Domain defined in
template, and if a domain is instantiated from such a template, then encryption can be Enabled/Disabled at Zone
and Subnet level when they are added to the instance.

Fig. 7.9: Enabling Encryption for Subnet

• Default setting for encryption on the Domain is ‘Disabled’ and ‘Inherited’ for Zone and Subnet.
• The encryption setting on the Subnet takes the highest precedence.

7.1. Group-Key IPsec 414

 VNS User Guide, Release 5.3.2

Note: Once the NSG successfully bootstraps and becomes active, it receives the SEK and the Seed from the KS
regardless of the encryption settings on the Domain, Zone, or Subnet (i.e. even if the flags on these objects are all set
to disable).

Once encryption is enabled at the subnet level, traffic to and from the secure subnet is always encrypted. Thus, even
if a subnet has encryption disabled, traffic to and from the subnet to a secure subnet is encrypted. As depicted in the
following Figure, VxLAN traffic is sent in the clear only when both subnets (S1 and S2) have encryption disabled.

Fig. 7.10: Encryption across Subnets

Create Secure L2 Domain

Similar to the Layer 3 service instantiation, for Layer 2 domains, encryption can be enabled or disabled at the Domain
template level. The Domain instance inherits the encryption settings from the Domain template and cannot be changed
at the Domain instance level. Enable encryption as follows:

1. In the Organization view, select Networks -> Layer 2 Domains , and select an L2 domain template.

2. In the center panel, select the template icon , and in the far right panel specify the encryption parameter
under the tab icon .

7.1. Group-Key IPsec 415

 VNS User Guide, Release 5.3.2

Fig. 7.11: Enabling Encryption for L2 Domain Template

7.1.7 VxLANoIPsec QoS Marking

For encrypted domains, the DSCP marking is always copied from the outer VxLAN header to the outer ESP header.
The DSCP marking on the outer VxLAN header is determined as a function of the Ingress QoS policy associated
with the VPort and the forwarding class to DSCP marking derived from the Network QoS in the VSC as explained in
VxLAN QoS Marking (page 416).

VxLAN QoS Marking

For traffic egressing to the overlay, the QoS marking on the customer packet header and outer VxLAN header is
determined as a function of the Forwarding Class Attributes of an Ingress QoS policy (associated with the VPort) as
follows:

7.1. Group-Key IPsec 416

 VNS User Guide, Release 5.3.2

Fig. 7.12: QoS Policy Forwarding Class Attributes

1. If the user checks the Rewrite Provider Class With Customer Class (Trust) option, then regardless of any
configured DCSP mapping tables or default forwarding class, the DSCP marking from the incoming customer
IP header is copied to the outer VxLAN header. As stated earlier, for encrypted domains, the DSCP marking is
also copied to the outer ESP header.
2. If the user checks Rewrite Customer Class With Provider Class option, then first the forwarding class is
determined depending on the following two scenarios:
1. There is no DSCP Mapping Table: In this case the default forwarding class (FC) is used.

7.1. Group-Key IPsec 417

 VNS User Guide, Release 5.3.2

2. There is a DSCP Mapping Table, then the FC is determined per the Ingress QoS.
Once the FC is determined, then the DSCP marking is derived from the Network QoS (page 418) in the
VSC. The derived DSCP marking is then copied to the customer IP header and to the outer VxLAN
header, as well as the outer ESP header (for encrypted domains).
3. If the user does not check either Rewrite option, then the customer packet header’s DSCP marking will not be
re-written. Only the outer VxLAN header and the outer ESP header (in the case of encryption) will be re-written
with the derived DSCP marking using default FC or from DSCP Mapping table.
4. If there is no Ingress QoS policy attached to the VPort, then DSCP in outer VxLAN header will be 00.

Network QoS Configuration on VSC

There is a default Network QoS on the VSC that may be replaced by a user created QoS. Run the following command
to check which network QoS policy is in effect on the VSC (in this example, the default QoS is in effect):

*A:vsc>config>vswitch-controller>open-flow# info detail

----------------------------------------------
tls-profile "of-tls-profile"
hold-time 15
echo-time 5
gw-hold-time 3
gw-echo-time 1
network-qos-policy 1
----------------------------------------------

Create a new policy as follows:

Example

*A:vsc# configure qos network 2 create

*A:vsc>config>qos>network>egress>$ fc "l1"
*A:vsc>config>qos>network>egress>fc$ dscp-in-profile "ef"
*A:vsc>config>qos>network>egress>fc$ dscp-out-profile "cp63"

Check the new policy as follows:

Example

*A:vsc>config>qos>network>egress>fc$ info
-------------------------
dscp-in-profile ef
dscp-out-profile cp63
-------------------------

The DSCP values can be:

“be” “ef” “cp1” “cp2” “cp3” “cp4” “cp5” “cp6” “cp7” “cp9” “cs1” “cs2” “cs3” “cs4” “cs5” “nc1” “nc2”
“af11” “af12” “af13” “af21” “af22” “af23” “af31” “af32” “af33” “af41” “af42” “af43” “cp11” “cp13”
“cp15” “cp17” “cp19” “cp21” “cp23” “cp25” “cp27” “cp29” “cp31” “cp33” “cp35” “cp37” “cp39”
“cp41” “cp42” “cp43” “cp44” “cp45” “cp47” “cp49” “cp50” “cp51” “cp52” “cp53” “cp54” “cp55”
“cp57” “cp58” “cp59” “cp60” “cp61” “cp62” “cp63”

7.1. Group-Key IPsec 418

 VNS User Guide, Release 5.3.2

7.1.8 Interworking with NAT-T

When the NSGs are behind a NAT device, the ESP packets traverse the NAT device seamlessly. As part of the Full
Cone NAT control plane communication, each NSG will set up two DTLS sessions to the VSC using UDP ports 4789
(for VxLAN) and 4500 (for IPsec) as source ports.

7.1.9 CLI Command Sample Output

{} - optional filter
<> - required filter

ovs-appctl ipsec/list-sek {customer-id}

Seed Encryption Key (SEK) entries:
ID Create Time Start Time Time left(sec) Enc Algo Auth Algo CustomerID
--------------------------------------------------------------------------------------
˓→----------------------------

1475185460226 2016-09-29 21:44:20 2016-09-29 21:45:20 75496 AES_256_CBC HMAC_SHA1

˓→10005

1475185459352 2016-09-29 21:44:19 2016-09-29 21:45:19 75495 AES_256_CBC HMAC_SHA1

˓→10004

ovs-appctl ipsec/list-seed {customer-id}

Seed entries:
ID Create Time Start Time Time left(sec) CustomerID
--------------------------------------------------------------------------------------
1475185460361 2016-09-29 21:44:20 2016-09-29 21:45:20 3436 10005
1475185459652 2016-09-29 21:44:19 2016-09-29 21:45:19 3435 10004

ovs-appctl ipsec/list-sa {customer-id}

IPSec SAs:
SPI Seed ID Enc Algo Auth Algo Soft life(sec) Hard life(sec) CustomerID
--------------------------------------------------------------------------------------
˓→------------------------

0x485e500f 1475185460361 AES_128_CBC HMAC_SHA1 193 373 10005

0x2f2dd384 1475185459652 AES_128_CBC HMAC_SHA1 192 372 10004

ovs-appctl ipsec/list-policies {customer-id}

IPSec policies:
VxLAN src VxLAN dst ESP local ESP remote UDP dport Customer-ID VRFs
--------------------------------------------------------------------------------------
˓→----------------------

109.37.199.86 100.71.132.3 10.15.1.254 10.18.3.254 4500 10004 (21c90fad)

109.37.199.86 100.71.132.3 10.18.1.254 10.18.3.254 4500 10004 (21c90fad)
109.37.199.86 100.71.132.3 10.15.1.254 10.15.3.254 4500 10004 (21c90fad)
109.37.199.86 100.71.132.3 10.18.1.254 10.15.3.254 4500 10004 (21c90fad)

ovs-appctl ipsec/list-xfrm-state <remote-datapth-id> {customer-id}

Src Dst Dir SPI SKB mark Customer-ID
------------------------------------------------------------------------------
10.18.3.254 10.15.1.254 In 0xbc21789e 0x0 10004
10.18.3.254 10.15.1.254 In 0xec36f698 0x0 10004
10.18.3.254 10.15.1.254 In 0xdfa6c0ec 0x0 10004
10.18.3.254 10.15.1.254 In 0x929ac0d1 0x0 10004
10.18.3.254 10.15.1.254 In 0xb1f8ab6a 0x0 10004
10.18.3.254 10.15.1.254 In 0x7db6aa03 0x0 10004

7.1. Group-Key IPsec 419

 VNS User Guide, Release 5.3.2

10.18.3.254 10.15.1.254 In 0xcca51b81 0x0 10004

10.18.3.254 10.15.1.254 In 0x61081f2c 0x0 10004
10.15.1.254 10.18.3.254 Out 0x66986e88 0x49000100 10004
10.15.1.254 10.18.3.254 Out 0x6d788943 0x49000120 10004
10.15.1.254 10.18.3.254 Out 0x1b496172 0x49000128 10004
10.15.1.254 10.18.3.254 Out 0x73f470fe 0x49000148 10004
10.15.1.254 10.18.3.254 Out 0xcff35646 0x49000188 10004
10.15.1.254 10.18.3.254 Out 0xdfca6776 0x490001b8 10004
10.15.1.254 10.18.3.254 Out 0x9eefadb0 0x490001c0 10004
10.15.1.254 10.18.3.254 Out 0x66473f69 0x490001e0 10004

ovs-appctl ipsec/list-svc-customer {customer-id}

Customer to SKB mark association table:
SVC-ID Customer-ID SKB mark
--------------------------------------------
4ec21ae9 10004 0x100

ovs-appctl ipsec/show-stats <remote-datapath-id> <customer-id>

IPsec traffic stats for remote NSG:115.146.99.120
Encrypted packets: 0
Encrypted bytes: 0
Decrypted packets: 0
Decrypted bytes: 0
Error stats
Replay errors: 0
Authentication errors: 0

7.1. Group-Key IPsec 420

 VNS User Guide, Release 5.3.2

7.2 IKE IPsec

• Overview (page 421)

• Terminology (page 423)
• Use Cases (page 423)
– Feature Interworking Use Cases (page 423)
• Architecture Overview (page 424)
– Forwarding Behavior (page 424)
– IKE Protocol Overview (page 425)
• HTTP Ping for IKE Tunnels (page 425)
• IKE Security Configuration (page 425)
– High Level Workflow (page 425)
– Detailed Workflow (page 427)

* Create IKE Encryption profile (page 427)

* Create IKE Gateway (page 431)
* Create IKE Pre Shared Keys (page 433)
* Create IKE Gateway Profile (page 434)
* Create NSG Gateway Connection (page 435)
• IKE Connection Redundancy in 1:Many Topology (page 437)
• CLI Commands (page 438)

7.2.1 Overview

The Internet Key Exchange (IKE) capability is added to the existing group-key implementation of IPsec for inter-
operability with 3rd party Gateways that support the IKE protocol (specifically IKEv2). The feature enables secure
connection of remote branch location, over Internet or trusted 3rd party network, to a centralized data center with an
IKE Gateway.
The IKE Gateways may be:
• 7x50 IPsec with ISA cards
• Cisco ASR
• Public cloud Gateways (e.g. AMZ VPN gateways)
The recommended target topology is hub-and-spoke whereby communications between an NSG and a remote gateway
(NSG or 3rd party gateway) is encrypted using IKE IPsec. Any NSG to NSG communication will always use group-
key for encrypting traffic.

7.2. IKE IPsec 421

 VNS User Guide, Release 5.3.2

Fig. 7.13: IKE IPsec Topology

The following topologies are supported:

• NSG as 1-to-1 Initiator or Responder
• NSG as 1-to-Many Hub, where the NSG can only be a Responder

Fig. 7.14: IKE Connection Topology

The IKE Connection Topology figure illustrates an NSG with dual uplinks. The following connections are depicted:
• NSG as 1:1 Initiator to gateway GW1. Local and Remote subnets are configured via the VSD.
• NSG as 1:1 Responder to gateway GW2. Local and Remote subnets are configured via the VSD.
• NSG as 1:Many Responder on uplink port 1. Gateways GW3, GW4, and GW5 are initiators. Only Local subnet
is configured via the VSD. Remote subnet is Any and a remote subnet is proposed when a connection is initiated.
• NSG as 1:Many Responder on uplink port 2. Gateways GW6, GW7, and GW8 are initiators. Only Local subnet
is configured via the VSD. Remote subnet is Any and a remote subnet is proposed when a connection is initiated.

7.2. IKE IPsec 422

 VNS User Guide, Release 5.3.2

7.2.2 Terminology

• Tunnel-based IPsec : ESP based implementation that relies on IKE for authentication, key generation and dis-
tribution.
• Group-key based IPsec: ESP based implementation that relies on KS entity to handle end-point authentication
(member registration), key (re)generation and distribution.
• Security Association (SA): A policy that governs encryption and encapsulation parameters between a set of
IPsec peers belonging to a specific domain. Multiple domains on a single NSG will have different SAs.

7.2.3 Use Cases

Example 1: An IP/MPLS based IPVPN implemented using 7x50. Off-net sites are connected via IPsec to a designated
7x50 hosting an IPsec-ISA. On the 7x50 GW, IPsec tunnels originating from remote NSGs are terminated and customer
payload is mapped to the respective VRF for access to the on-net IP/MPLS VPN.
Example 2: An Amazon deployed VPN gateway (IKE-based IPsec terminator) implemented to connect a VPC to
an Enterprise HQ site. NSG functioning as the HQ WAN router is required to terminate traffic from remote NSGs
(located at branches) as well as traffic source/terminated by AMZ VPC.
Example 3: Existing Cisco router deployed at a location whereby NSGs need to receive/send traffic to such location
over encrypted connection.

Note: Both IKE and group-key tunnels can co-exist within the same NSG/Domain.

Fig. 7.15: Group Key and IKE IPsec in a Domain

Feature Interworking Use Cases

• Encryption of mirrored traffic when a mirrored destination is behind an IKE gateway. Local subnet must be set
to Any (described later). The uplink IP is used as the source for mirrored traffic.
• BGP control packets encrypted for BGP peer behind an IKE gateway on the uplink. Currently, only one BGP
peer is allowed. Both local and remote subnets must be set to Any (described later).
• IKE over PPPoE is supported.

7.2. IKE IPsec 423

 VNS User Guide, Release 5.3.2

7.2.4 Architecture Overview

The following VNS services are supported with IPsec:

• Non-overlay based services: Domains that do not use any tunnels and send directly to the underlay without
additional encapsulation
• L3 domains
The VNS components are leveraged as follows:
• VSD hosts IKE policies.
• VSC passes the policies transparently to NSG.
• NSG performs IKE control-plane procedures along with tunnel establishment and traffic encryption (data-plane).
IKE based IPsec tunnels are applicable only for traffic egressing the NSG uplink ports. In contrast to group-key based
implementation, IKE initiated tunnels support IPoESP encapsulation (i.e. no VXLAN encapsulation).

Forwarding Behavior

Fig. 7.16: IKE Forwarding Behavior

LAN to WAN IKE IPsec traffic forwarding behavior is as follows:

• If routes exist in the overlay, then traffic is forwarded to overlay.
• If no routes in overlay then:
– If Underlay Support is disabled then traffic is dropped.
– If Underlay Support is enabled, then traffic is forwarded to underlay.

* If routes exist with next-hop using IKE tunnel, then forward over IPsec tunnel.
* If not, then forwarding is per NAT/PAT routing.

Note: Underlay Support must be enabled on the subnet for IKE IPsec tunneling.

• If all rules fail then traffic is dropped unless there is a default route.
It is assumed that subnets that are behind IKE gateways are not present in the overlay. Otherwise the overlay policy
evaluation will take precedence. Also domain/VRF context is not maintained when traffic is forwarded to an IKE
IPsec tunnel.

7.2. IKE IPsec 424

 VNS User Guide, Release 5.3.2

Note: IKE Gateways are associated with a specific NSG Network Port. There is no fail-over to a second network port
if the port with Gateway association fails.

IKE Protocol Overview

Internet Key Exchange (IKE) protocol is a negotiation protocol that lets two communicating peers to agree and es-
tablish security associations (SA). IKE provides strong authentication of both peers and derives unique cryptographic
session keys.
Besides authentication and key material, IKE also provides the means to exchange configuration information and to
negotiate IPsec SAs (also referred to as child SAs). IPsec SAs define which network traffic is to be secured and also
describe the algorithms and keys used to encrypt and authenticate the traffic.
The Internet Security Association and Key Management Protocol (ISAKMP) and IPsec tunneling standards are used
to build and manage SAs in two phases:
• Phase 1 creates the first tunnel, which protects later ISAKMP negotiation messages to establish the IKE Secu-
rity Association (IKE-SA). Once the IKE-SA is created, it can be used for sending authenticated notification
messages, reliable dead-peer detection, and inexpensive creation of additional child-SAs.
• Phase 2 creates the tunnel that protects data. Specifically phase 2 establishes IPsec SAs. IPsec SAs are unidi-
rectional. Each SA consists of values such as destination address, a security parameter index (SPI), the IPSec
transforms used for that session, security keys, and additional attributes such as IPSec lifetime.

7.2.5 HTTP Ping for IKE Tunnels

You can configure a performance monitor to perform an HTTP ping on an IKE IPsec tunnel to monitor reachability
to a third party IKE gateway. They can be used to determine whether an active IKE tunnel has failed and to perform
a switchover to a standby tunnel, if required. These performance monitors are similar to the one-way active measure
protocol probes used for AAR, but they do not require an NSG as the destination target.
HTTP ping for IKE tunnels is supported as part of the SaaS routing solution. See SaaS performance and reachability
monitoring (page 562) for more information.

7.2.6 IKE Security Configuration

IKE security configuration has an Enterprise scope, and is typically provisioned by an enterprise administrator. The
high level workflow is presented as an overview, followed by a more detailed step-by-step workflow.

High Level Workflow

1. Create IKE Encryption profile(s) to specify the IKE policy parameters. This is optional because when an
Enterprise is created, a default IKE Encryption profile is created per enterprise. The default profile cannot be
renamed or deleted. However, the various parameter values in the default profile may be edited. The parameters
are grouped as follows:
1. ISKAMP (Internet Security Association and Key Management Protocol) which is used by IKE for establishing
the IKE-SA.
2. IPsec which is used for the traffic encryption
3. Peer Detection timers that become active as soon as the IKE-SA is established. These are used for dead peer
detection.

7.2. IKE IPsec 425

 VNS User Guide, Release 5.3.2

2. Create IKE Gateway(s) to define and identify the 3rd party Gateway device(s). Once created, a Gateway may
be updated even if associated with an NSG port instance. However, a Gateway may only be deleted if there are
no NSG port instance associations.

Note: To support the 1-to-Many topology, an Any_Gateway configuration object is provided by default. The
Any_Gateway is used in a 1:Many Gateway Connection to allow any remote gateway to connect to the Hub NSG.
By default the right subnet of the Any_Gateway is 0.0.0.0/0.

Fig. 7.17: IKE Any_Gateway Object

Once a IKE Gateway is created, attach IKE Gateway Subnet(s) (also referred to as right subnet(s)). These represent
the routes on the IKE Gateway side.
3. Define IKE Pre Shared Keys. Currently only the Pre Shared Key (PSK) authentication mode is supported. In
the future RSA Signature mode will also be supported. User is required to create PSK(s) that will be used for
peer authentication.
4. Create IKE Gateway Profile
1. Associate an Encryption Profile. If a profile is not associated, then the default profile will be used.
2. Associate IKE Gateway. Use the Any_Gateway if a 1:Many Gateway Connection is required.
3. Associate Authentication (optional). This parameter provides a Gateway level scope for the authentication mode
that is inherited by every connection with which the profile is associated. It is always possible to override the
mode at the Gateway connection level.
5. Create NSG Gateway Connection. It is the connection between the IKE Gateway and the NSG network port.
1. Associate an IKE Gateway Profile.
2. Associate Authentication (optional). If specified, then this will take precedence if Authentication is also in the
IKE Gateway Profile.
3. The user may override authentication associated in step 5(a) and/or 5(b), and enter a completely new PSK in the
NSG Gateway Connection dialog. If specified, then the new PSK will be used for authentication.
4. Specify the NSG role as Initiator or Responder
6. Attach Local Subnets
Once the NSG Gateway Connection is created, attach Local Subnet(s). Only those subnet(s) for which
underlay support has been enabled will be available for attachment.

Note: For a given uplink port on an NSG, you should not create gateway connections with the same gateway profile.

7.2. IKE IPsec 426

 VNS User Guide, Release 5.3.2

7. Create a Performance Monitor for HTTP ping.

1. Ensure performance management is enabled for the organization.
2. Create a performance monitor of type HTTP.
3. Assign NSGs to the performance monitor.

Detailed Workflow

Create IKE Encryption profile

In the enterprise view, navigate to Infrastructure tab and then in the left panel, select IKE Gateway Profiles –>
IKE Encryption Profiles . Click on to add a new profile.

Fig. 7.18: Create IKE Encryption Profile (default values displayed)

Enter a Name and Description and then configure the parameters in the following tabs of the encryption profile:

a. ISKAMP Parameters:

• Diffie-Hellman Group Identifer for peers to derive a shared secret without transmitting it to each
other.
• Authentication Mode to establish identity of the peers.
• Hash Algorithm to ensure the identity of the sender, and to ensure that the message has not been
modified in transit.

7.2. IKE IPsec 427

 VNS User Guide, Release 5.3.2

• Encryption Algorithm to protect the data and ensure privacy. It specifies the symmetric encryption
algorithm that protects data transmitted between two peers.
• Encryption Key Life Time is a duration for how long the encryption key is used before replacing
it.
• Perfect Forward Secrecy (PFS) flag which determines the rekeying of the child SAs when the
IKE-SA is rekeyed. Enabling PFS keeps the rekeying in Phase 1 and Phase 2 in sync. PFS must
also be enabled on the IKE Gateway. Enabling PFS will have some performance impact so should
be used with caution.
The options for the ISKAMP attributes are listed in the following table.

Attribute Value

Diffie Hellman Group Identifier Group 1 768-bit DH

Group 2 1024-bit DH
Group 5 1536-bit DH
Group 14 2048-bit DH
Group 15 3072-bit DH
Group 16 4096-bit DH
Group 17 6144-bit ECDH
Group 18 8192-bit ECDH

Authentication Mode Pre-Shared Key

Hash Algorithm SHA1

SHA256

Encryption Algorithm TDES

AES128
AES192
AES256

Encryption Key Lifetime Min=60, Max=86400

Enable PFS True or False

As part of the peer negotiation, a match exists when the security policies of the two peers contain the same encryption,
hash, authentication, and Diffie-Hellman parameter values, and when the remote peer policy specifies a lifetime less
than or equal to the lifetime in the policy the initiator sent. If the lifetimes are not identical, then the shorter lifetime is
applied. If no acceptable match exists, ISAKMP refuses negotiation and the IKE-SA is not established.

7.2. IKE IPsec 428

 VNS User Guide, Release 5.3.2

b. IPsec Parameters:

Fig. 7.19: Create IKE Encryption Profile - IPsec (default values displayed)

• Encryption Algorithm for protecting the data transmitted between two peers.
• Authentication Algorithm to ensure the identity of the sender, and to ensure that the message has not been
modified in transit.
• SA Life Time is a duration for how long the phase 2 encryption key is used before replacing it.
• SA Replay Window Size is required for replay detection. This parameter applies only if anti-replay is enabled
in the IKE Gateway Profile.
Although not an explicit configurable option via the user interface, Don’t Fragement (DF) capability is supported for
the uplink underlay network. NSG copies the DF bit from the payload to the ESP header. If DF is 1 and ESP packet
size exceeds the port MTU, then it is dropped. If DF is 0 and packet size exceeds port MTU, then it is fragmented.

7.2. IKE IPsec 429

 VNS User Guide, Release 5.3.2

Attribute Value

Encryption Algorithm TDES

AES128
AES192
AES256
NULL

Authentication Algorithm SHA1

SHA256
SHA 512
MD5

SA Lifetime Min=60, Max=86400

SA Replay Window Size 128

c. DPD Parameters

Fig. 7.20: Create IKE Encryption Profile - DPD (default value displayed)

The DPD Mode can be either On-Demand or Reply-only.

• Reply-only is the default option. For this option, no additional timer configuration required.
• If NSG role is initiator, then On-Demand is the only option that applies.
When on-demand is selected and NSG doesn’t receive traffic for a second, then a keep alive message is sent to the
peer every N seconds, where N is the user configured DPD interval.
• For IKEv1, if there is no traffic or response from the peer for a duration of three times the configured DPD
interval, then NSG will consider peer as dead and clear all the tunnels. After DPD timeout, the NSG starts
initiating IKEv1 negotiations forever.

7.2. IKE IPsec 430

 VNS User Guide, Release 5.3.2

• For IKEv2, if there is no response to the keep alive, then a keep alive will be sent every 5 seconds for 5 times. If,
after the retries, the NSG doesn’t receive any traffic or response to the keep alives, then NSG will consider peer
as dead and clear all the tunnels. Once the retries have completed, the NSG starts initiating IKEv2 negotiations
forever.

Mode Interval

On-Demand Min=10, Max=120

Note: It is assumed that the DPD attributes are also configured on the peering IKE Gateway to match the setting on
the NSG.

Create IKE Gateway

Navigate to Infrastructure tab and then in the left panel, select IKE Gateway Profiles –> IKE Gateways .
Click on to add a new gateway.

7.2. IKE IPsec 431

 VNS User Guide, Release 5.3.2

Fig. 7.21: Create IKE Gateway

• Enter Name and Description for the IKE Gateway

• Enter IP address (optional). However, if the Gateway will be associated with an NSG that has the initiator role,
then the IP Address is mandatory.
• Specify Version as IKE Version 1 or IKE Version 2.
• Specify the Mode which only applies to IKE Version 1, and can be either Main or Aggressive. For IKE Version
2, the Mode defaults to None.

Specify Remote Subnets

Specify the subnet(s) behind the IKE Gateway. These are referred to as remote (or right) subnets. In the Remote
Subnets panel to the right of a selected IKE Gateway, add new subnets by clicking on at the bottom of the panel.

7.2. IKE IPsec 432

 VNS User Guide, Release 5.3.2

Fig. 7.22: Add Remote Subnet

Note:
• For BGP control packets to be encrypted (to a BGP peer behind the IKE gateway), the remote subnet must be
set to 0.0.0.0/0. It is also required that the local subnet be set to Any (described later in this chapter).
• For mirrored traffic to be encrypted (when a mirrored destination is behind the IKE gateway), the remote subnet
must be set to 0.0.0.0/0.

Create IKE Pre Shared Keys

Navigate to Infrastructure tab and then in the left panel, select IKE Gateway Profiles –> IKE Pre Shared Keys
. Click on to add new PSK(s).
Peer authentication is a part of the IKE IPsec setup process. Currently only PSK mode is supported as the authentica-
tion mode.

Fig. 7.23: Create IKE PSK

• Enter Name and Description for the PSK.

• Enter an alphanumeric string for the Unencrypted Pre Shared Key and re-enter to confirm.
User may create multiple PSKs that can be associated with IKE Gateway peers as needed. Same PSK must be
configured on the peer.

7.2. IKE IPsec 433

 VNS User Guide, Release 5.3.2

Create IKE Gateway Profile

Navigate to Infrastructure tab and then in the left panel, select IKE Gateway Profiles . Click on to create new
Gateway Profiles.

Fig. 7.24: Create IKE Gateway Profile

• Enter Name and Description for the profile

• Associate an Authentication Method (optional) by clicking on the link .

• Associate a Gateway

Fig. 7.25: IKE Gateway ID Types

• Enable or disable Anti-Replay. When enabled, receiver rejects obsolete or repeated packets.
• When anti-replay is enabled, a Service Class must be selected or the default forwarding class H will be applied.
The selected service class will be used for classification and DSCP remarking of the outer header of all traffic
on IKE tunnels. It will be also be used for Egress QoS.

Note: When anti-replay is enabled:

• Any Ingress QoS classifications and DSCP to Forwarding Class markings are ignored for IKE tunnel traffic
only. Classification and markings for overlay traffic is not affected.
• For BGP peering, the BGP control plane traffic will be classified to the Forwarding Class configured in the
gateway profile.

7.2. IKE IPsec 434

 VNS User Guide, Release 5.3.2

• Select a Gateway Identifier Type for creating an identifier from one of the following types:
– ID_IPV4_ADDR: An IPv4 Address. It is the default for the IKE Gateway and uses the configured IP
address of the gateway as the Gateway Identifier. However, if IP address of gateway is not specified, then
some type of identifier is required.
– ID_FQDN: a fully qualified domain name string
– ID_RFC822_ADDR: an email address string
– ID_KEY_ID: an opaque octet stream. On the NSG side, it is the default and uses the NSG Datapath ID in
a hexadecimal format.
• Enter a Gateway Identifer corresponding to the selected type.
• Associate an Encryption Profile. This is optional because the default will applied if not specified.

Create NSG Gateway Connection

Navigate to Infrastructure -> Network Service Gateways -> NSG Instance –> Network Port (Port 1 or Port 2)
-> Untagged VLAN –> Gateway Connections panel. Click on to create new Gateway Connection.

Fig. 7.26: Create NSG Gateway Connection

• Enter a Name for the connection

• Specify the NSG Connection Priority which defines the forwarding priority if the same subnet is behind two
different IKE Gateways. The tunnel is established with both Gateways, and the priority is used to forward the
traffic on the tunnel with the lower priority number. If not specified, then VSD will assign a priority.

7.2. IKE IPsec 435

 VNS User Guide, Release 5.3.2

Note:
• If for a given local/remote subnet combination, there is more than one gateway connection, then priority based
forwarding will ensure that encrypted traffic will flow over the highest priority active connection.
• On-demand DPD is required for detecting connection failures and for switching traffic across tunnels; such as,
traffic will switch to the lower priority connection if the remote gateway of a high priority connection is declared
dead, and switch back to the higher priority tunnel when the connection is restored.
• IKE Gateways must have a unique priority if associated with the same NSG. Priority value must be between 1
and 16000.

• Select the NSG Identifier Type from the same set as described for the Gateway Identifier Type. ID_KEY_ID
is the default.
• Enter an NSG Identifier. Default is NSG’s Datapath ID in a hexadecimal format and corresponds to the
ID_KEY_ID type.
• Select the NSG Role as either Initiator/Responder. It specifies if the NSG initiates an IKE IPsec connection or
only responds to connection requests from IKE Gateways.

Attention:
• If the NSG is behind a NAT device, then the NSG must be the initiator. When NSG is initiator, the IP address
of the associated IKE Gateway is required.
• If NSG is acting as a hub, then the NSG can only be a responder. For an NSG to be an initiator, it must be a
spoke.
• Once configured, NSG role cannot be changed from responder to initiator and vice versa.
• There can be a maximum of one 1:Many Gateway Connection per uplink.
• There may be multiple 1:1 IKE Gateways associated per NSG port instance, and the Gateways may be
associated/de-associated at any time

• The All local subnets flag when enabled implies that every configured connection with a source IP in the local
subnet(s) will be encrypted.

Note: For interworking with BGPv4, if there is a BGP peer behind the IKE gateway, then the All local subnets flag
must be enabled. The remote subnet must also be set to Any.

• Associate a Gateway Profile with the NSG.

• Associate an Authentication Mode (optional) with the NSG. The Authentication Mode (if associated) takes
precedence over any authentication that may be included in the Gateway Profile.
User may override any existing authentication (in Authentication Profile and/or in Gateway Profile) by providing a new
PSK in the dialog. Regardless, at least one authentication association must be included as part of the NSG Gateway
Connection.

7.2. IKE IPsec 436

 VNS User Guide, Release 5.3.2

Fig. 7.27: Enter PSK for Authentication

Attach Local Subnets

Fig. 7.28: Attach Local Subnet

Once the NSG Gateway Connection is created, attach Local Subnet(s). Subnets that are bound to the NSG through
VPort association are candidates for attachment. The VPorts do not have to be resolved, just associated. From the
candidate list of subnets, only those subnet(s) for which underlay support has been enabled will be available for
attachment.

7.2.7 IKE Connection Redundancy in 1:Many Topology

In a 1:Many scenario, if multiple initiators propose the same (or an overlapping subnet) traffic selectors, then there will
be outbound and inbound traffic only on the tunnel that gets initiated first. Other tunnels will be marked as inactive
since the policy will only be applied to the tunnel that gets established first.
For example, as shown in the Figure below, the NSG is configured with two 1:Many connections (one on each uplink
port), using a local subnet of 20.0.5.0/24 for each connection. On each connection, the various gateways initiate a
connection and propose the same remote subnet 3.1.1.0/24. This depicts a completely redundant case with identical
local/remote subnets.

7.2. IKE IPsec 437

 VNS User Guide, Release 5.3.2

Fig. 7.29: IKE Connection Topology with Redundancy

If we assume that the 1:Many connection on Port1 is created with a higher priority than the 1:Many connection on
Port2, then the following is the expected behavior:
Since this is a redundant scenario, the tunnel will be established on both uplinks, but the connection priority is used to
forward on the tunnel on the higher priority connection. However, since both ports have 1:Many connection, on each
port, only the tunnel that gets established first (of all the initiators) will be active. In effect, in the depicted scenario
there will be outbound and inbound traffic only on the tunnel that gets initiated first on port1 (since we assume the
connection has higher priority).

7.2.8 CLI Commands

Following Show commands are supported (the name argument is optional):

nuage-nsg-ike-cli show enc-profile <enc-profile-name>
• shows a table with information about the all encryption profiles setup on the NSG.
nuage-nsg-ike-cli show gw-connections <gw-connection-name>
• shows the table with information about all the gateway connections originating from the NSG.
nuage-nsg-ike-cli show ike-sa <gw-connection-name>
• shows the list of ike-sa that are loaded for each gateway connection.
nuage-nsg-ike-cli show ipsec-sa <gw-connection-name>
• shows the list of ipsec-sa that are loaded for each gateway connection.
nuage-nsg-ike-cli show ip-xfrm-state <gw-connection-name>
• shows the table with information about SPIs, IPs and direction for every tunnel configured in kernel.
nuage-nsg-ike-cli show ip-xfrm-policy <gw-connection-name>
• shows the table with information about local and remote IPs,subnets for every policy configured in kernel.
nuage-nsg-ike-cli show ipsec-stats
• displays stats for all gateway connections. Display could be delayed by maximum of 5 seconds since
polling is every 5 seconds.

7.2. IKE IPsec 438

 VNS User Guide, Release 5.3.2

nuage-nsg-ike-cli show traffic-selectors

• displays all gateway connections for each set of local and remote subnet, and identifies which is Active
(Primary)
nuage-nsg-ike-cli show tunnel-status-summary
• displays tunnel attributes and phase 1 and phase 2 status for all connections
nuage-nsg-ike-cli show remote-initiators
• displays remote initiator connection details
Following Count commands are supported:
nuage-nsg-ike-cli count enc-profile
• displays count of configured encryption profiles.
nuage-nsg-ike-cli count gw-connections
• displays count of gateway connections configured.
nuage-nsg-ike-cli count ipsec-tunnels
• displays count for gateway tunnels that are up and down.
Sample CLI output is shown below:

*A:vsc# tools vswitch <nsg-ip> command "nuage-nsg-ike-cli show enc-profile"

Name: My_Ikev2_Enc_Prof ID: 73874030-f822-4752-a19d-064509e2e96f

ike-auth-mode: PRE_SHARED_KEY ike-auth-algo: SHA256 ike-enc-algo: AES256
ike-dh-group: GROUP_2_1024_BIT_DH ipsec-pfs: True
ike-sa-lifetime: 86400 ike-dpd-interval: 0 ike-dpd-retry-interval: 0
ipsec-auth-algo: HMAC_SHA256 ipsec-enc-algo: AES128
ipsec-sa-lifetime: 3600 ipsec-df-bit: False ipsec-pre-frag: True
Name: Default IKE Encryption Profile ID: 8a1eb94f-bebd-4b89-87c4-c2bf77cef35e
ike-auth-mode: PRE_SHARED_KEY ike-auth-algo: SHA256 ike-enc-algo: AES256
ike-dh-group: GROUP_5_1536_BIT_DH ipsec-pfs: False
ike-sa-lifetime: 1200 ike-dpd-interval: 0 ike-dpd-retry-interval: 0
ipsec-auth-algo: HMAC_SHA256 ipsec-enc-algo: AES256
ipsec-sa-lifetime: 3600 ipsec-df-bit: False ipsec-pre-frag: True

--------------------------------------------------------------------------------------
˓→--------------------------

*A:vsc# tools vswitch <nsg-ip> command " nuage-nsg-ike-cli show gw-connections"

Name: My_Ikev2_Gw_Conn_4 ID: 519658df-2fb7-4596-a120-c271aacabfe9

enc-profile-id: 73874030-f822-4752-a19d-064509e2e96f
state: Modified cert-id: none
nsg-role: INITIATOR port-vlan: port1-vlan0
local-id-type: ID_KEY_ID local-id: fb1bbed6-3663-4c17-8ceb-a7fa3379ac0a
˓→local-ip: 10.15.1.254

remote-id-type: ID_IPV4_ADDR remote-id: 51.1.1.1 remote-ip: 51.1.1.1

local-subnets: 20.0.1.0/24 remote-subnets: 3.2.1.0/24
Name: My_Ikev2_Gw_Conn_2 ID: 95429ce4-2392-4d6d-bca4-d789fe936497
enc-profile-id: 73874030-f822-4752-a19d-064509e2e96f
state: New cert-id: none
nsg-role: INITIATOR port-vlan: port1-vlan0
local-id-type: ID_KEY_ID local-id: fb1bbed6-3663-4c17-8ceb-a7fa3379ac0a
˓→local-ip: 10.15.1.254

7.2. IKE IPsec 439

 VNS User Guide, Release 5.3.2

remote-id-type: ID_IPV4_ADDR remote-id: 21.1.1.1 remote-ip: 21.1.1.1

local-subnets: 20.0.0.0/24 remote-subnets: 2.2.1.0/24
Name: My_Ikev2_Gw_Conn_3 ID: d063bf7a-ff69-4ecd-b3a4-4cab674d0cf5
enc-profile-id: 73874030-f822-4752-a19d-064509e2e96f
state: Modified cert-id: none
nsg-role: INITIATOR port-vlan: port1-vlan0
local-id-type: ID_KEY_ID local-id: fb1bbed6-3663-4c17-8ceb-a7fa3379ac0a
˓→local-ip: 10.15.1.254

remote-id-type: ID_IPV4_ADDR remote-id: 50.1.1.1 remote-ip: 50.1.1.1

local-subnets: 20.0.1.0/24 remote-subnets: 3.1.1.0/24
Name: My_Ikev2_Gw_Conn_1 ID: ef09768a-d9b9-487a-9088-55a5def86771
enc-profile-id: 73874030-f822-4752-a19d-064509e2e96f
state: New cert-id: none
nsg-role: INITIATOR port-vlan: port1-vlan0
local-id-type: ID_KEY_ID local-id: fb1bbed6-3663-4c17-8ceb-a7fa3379ac0a
˓→local-ip: 10.15.1.254

remote-id-type: ID_IPV4_ADDR remote-id: 20.1.1.1 remote-ip: 20.1.1.1

local-subnets: 20.0.0.0/24 remote-subnets: 2.1.1.0/24

--------------------------------------------------------------------------------------
˓→--------------------------

*A:vsc# tools vswitch <nsg-ip> command "nuage-nsg-ike-cli show ike-sa"

ike-sa: 0e01ac14addff22b:c56c94e9e14cc77e gw-connection-name: My_Ikev2_Gw_Conn_1

status: ESTABLISHED local-id: fb1bbed6-3663-4c17-8ceb-a7fa3379ac0a local-ip: 10.
˓→15.1.254

remote-id: 20.1.1.1 remote-ip: 20.1.1.1

ike-enc-algo: AES256 ike-auth-algo: HMAC_SHA256 ike-dh-group: GROUP_2_1024_BIT_DH
state: established 2528s ago, rekeying in 79279s
child-sa: name:My_Ikev2_Gw_Conn_1 status: INSTALLED
in-spi:c052da51 out-spi:00080dae
ipsec-enc-algo: AES128 ipsec-auth-algo: HMAC_SHA256
local-subnet: 20.0.0.0/24 remote-subnet: 2.1.1.0/24
state: installed 330s ago, rekeying in 2767s, expires in 3270s
ike-sa: 2cb62be317699a13:8bc3ba6eafa03cbe gw-connection-name: My_Ikev2_Gw_Conn_2
status: ESTABLISHED local-id: fb1bbed6-3663-4c17-8ceb-a7fa3379ac0a local-ip: 10.
˓→15.1.254

remote-id: 21.1.1.1 remote-ip: 21.1.1.1

ike-enc-algo: AES256 ike-auth-algo: HMAC_SHA256 ike-dh-group: GROUP_2_1024_BIT_DH
state: established 2707s ago, rekeying in 80723s
child-sa: name:My_Ikev2_Gw_Conn_2 status: INSTALLED
in-spi:cefc114a out-spi:0006d02e
ipsec-enc-algo: AES128 ipsec-auth-algo: HMAC_SHA256
local-subnet: 20.0.0.0/24 remote-subnet: 2.2.1.0/24
state: installed 512s ago, rekeying in 2546s, expires in 3088s
ike-sa: 81e3d838193228bf:56bdbb5835f1aaf3 gw-connection-name: My_Ikev2_Gw_Conn_4
status: ESTABLISHED local-id: fb1bbed6-3663-4c17-8ceb-a7fa3379ac0a local-ip: 10.
˓→15.1.254

remote-id: 51.1.1.1 remote-ip: 51.1.1.1

ike-enc-algo: AES256 ike-auth-algo: HMAC_SHA256 ike-dh-group: GROUP_2_1024_BIT_DH
state: established 72372s ago, rekeying in 12326s
child-sa: name:My_Ikev2_Gw_Conn_4 status: INSTALLED
in-spi:c6049b24 out-spi:000510bf
ipsec-enc-algo: AES128 ipsec-auth-algo: HMAC_SHA256
local-subnet: 20.0.1.0/24 remote-subnet: 3.2.1.0/24
state: installed 2865s ago, rekeying in 111s, expires in 735s
ike-sa: df4c71ffffe1af91:bfa1ae43f7efb9f0 gw-connection-name: My_Ikev2_Gw_Conn_3

7.2. IKE IPsec 440

 VNS User Guide, Release 5.3.2

status: ESTABLISHED local-id: fb1bbed6-3663-4c17-8ceb-a7fa3379ac0a local-ip: 10.

˓→15.1.254
remote-id: 50.1.1.1 remote-ip: 50.1.1.1
ike-enc-algo: AES256 ike-auth-algo: HMAC_SHA256 ike-dh-group: GROUP_2_1024_BIT_DH
state: established 72374s ago, rekeying in 6847s
child-sa: name:My_Ikev2_Gw_Conn_3 status: INSTALLED
in-spi:c7aa8d89 out-spi:0002160f
ipsec-enc-algo: AES128 ipsec-auth-algo: HMAC_SHA256
local-subnet: 20.0.1.0/24 remote-subnet: 3.1.1.0/24
state: installed 74s ago, rekeying in 2881s, expires in 3526s

--------------------------------------------------------------------------------------
˓→--------------------------

*A:vsc# tools vswitch <nsg-ip> command "nuage-nsg-ike-cli show ipsec-sa"

Name:My_Ikev2_Gw_Conn_1 status: INSTALLED

in-spi:c052da51 out-spi:00080dae
ipsec-enc-algo: AES128 ipsec-auth-algo: HMAC_SHA256
local-subnet: 20.0.0.0/24 remote-subnet: 2.1.1.0/24
state: installed 370s ago, rekeying in 2727s, expires in 3230s
Name:My_Ikev2_Gw_Conn_2 status: INSTALLED
in-spi:cefc114a out-spi:0006d02e
ipsec-enc-algo: AES128 ipsec-auth-algo: HMAC_SHA256
local-subnet: 20.0.0.0/24 remote-subnet: 2.2.1.0/24
state: installed 552s ago, rekeying in 2506s, expires in 3048s
Name:My_Ikev2_Gw_Conn_4 status: INSTALLED
in-spi:c6049b24 out-spi:000510bf
ipsec-enc-algo: AES128 ipsec-auth-algo: HMAC_SHA256
local-subnet: 20.0.1.0/24 remote-subnet: 3.2.1.0/24
state: installed 2905s ago, rekeying in 71s, expires in 695s
Name:My_Ikev2_Gw_Conn_3 status: INSTALLED
in-spi:c7aa8d89 out-spi:0002160f
ipsec-enc-algo: AES128 ipsec-auth-algo: HMAC_SHA256
local-subnet: 20.0.1.0/24 remote-subnet: 3.1.1.0/24
state: installed 114s ago, rekeying in 2841s, expires in 3486s

--------------------------------------------------------------------------------------
˓→--------------------------

*A:vsc# tools vswitch <nsg-ip> command "nuage-nsg-ike-cli show ip-xfrm-state"

source destination spi direction count

--------------------------------------------------------------------------------
10.15.1.254 51.1.1.1 0x0001ec65 OUT 2(in:1, out:1)
0xc69a4e7f IN

10.15.1.254 21.1.1.1 0x0006d02e OUT 2(in:1, out:1)

0xcefc114a IN

10.15.1.254 50.1.1.1 0x0002160f OUT 2(in:1, out:1)

0xc7aa8d89 IN

10.15.1.254 20.1.1.1 0x00080dae OUT 2(in:1, out:1)

0xc052da51 IN

--------------------------------------------------------------------------------------
˓→--------------------------

7.2. IKE IPsec 441

 VNS User Guide, Release 5.3.2

*A:vsc# tools vswitch <nsg-ip> command "nuage-nsg-ike-cli show ip-xfrm-policy"

local_subnet remote_subnet local_ip remote_ip

˓→count

--------------------------------------------------------------------------------------
˓→--------------------

20.0.1.0/24 3.2.1.0/24 10.15.1.254 51.1.1.1

˓→3(in:1, out:1, fwd:1)

20.0.1.0/24 3.1.1.0/24 10.15.1.254 50.1.1.1

˓→3(in:1, out:1, fwd:1)

20.0.0.0/24 2.1.1.0/24 10.15.1.254 20.1.1.1

˓→3(in:1, out:1, fwd:1)

20.0.0.0/24 2.2.1.0/24 10.15.1.254 21.1.1.1

˓→3(in:1, out:1, fwd:1)

--------------------------------------------------------------------------------------
˓→--------------------------

*A:vsc# tools vswitch <nsg-ip> command "nuage-nsg-ike-cli show ipsec-stats"

--------------------------------------------------------------------------------------
˓→----------------------------------------------------------------

Gateway Name Local IP Remote IP encrypted_

˓→packets decrypted_packets encrypted_bytes decrypted_bytes
--------------------------------------------------------------------------------------
˓→----------------------------------------------------------------

My_Ikev2_Gw_Conn_6 10.15.1.254 50.1.1.1 0

˓→ 0 0 0
My_Ikev2_Gw_Conn_6 10.15.1.254 51.1.1.1 0
˓→ 0 0 0
My_Ikev2_Gw_Conn_6 10.15.1.254 52.1.1.1 583570
˓→ 116713 318045650 63608585
My_Ikev2_Gw_Conn_6 10.15.1.254 53.1.1.1 0
˓→ 0 0 0
My_Ikev2_Gw_Conn_6 10.15.1.254 54.1.1.1 0
˓→ 0 0 0
--------------------------------------------------------------------------------------
˓→----------------------------------------------------------------

*A:vsc# tools vswitch <nsg-ip> command "nuage-nsg-ike-cli show tunnel-status-summary"

--------------------------------------------------------------------------------------
˓→------------------------

Gateway Name Local IP Remote IP Phase1

˓→ Phase2
--------------------------------------------------------------------------------------
˓→------------------------

My_Ikev2_Gw_Conn_6 10.15.1.254 51.1.1.1 up

˓→ up
My_Ikev2_Gw_Conn_6 10.15.1.254 53.1.1.1 up
˓→ up
My_Ikev2_Gw_Conn_6 10.15.1.254 54.1.1.1 up
˓→ up
My_Ikev2_Gw_Conn_6 10.15.1.254 50.1.1.1 up
˓→ up

7.2. IKE IPsec 442

 VNS User Guide, Release 5.3.2

My_Ikev2_Gw_Conn_6 10.15.1.254 52.1.1.1 up

˓→ up
--------------------------------------------------------------------------------------
˓→------------------------

*A:vsc# tools vswitch <nsg-ip> command "nuage-nsg-ike-cli show remote-initiators"

--------------------------------------------------------------------------------------
˓→----------------------------------------------

Conn. ID Local IP Local TS Remote IP

˓→ Remote TS ID in Queue Oper State
--------------------------------------------------------------------------------------
˓→----------------------------------------------

de5d9a98-862e-4bc9-8957-e430acf53352 10.15.1.254 20.0.1.0/24 52.1.1.1

˓→ 3.1.1.0/24 1471902574 ACTIVE
de5d9a98-862e-4bc9-8957-e430acf53352 10.15.1.254 20.0.1.0/24 51.1.1.1
˓→ 3.1.1.0/24 1471902575 INACTIVE
de5d9a98-862e-4bc9-8957-e430acf53352 10.15.1.254 20.0.1.0/24 53.1.1.1
˓→ 3.1.1.0/24 1471902576 INACTIVE
de5d9a98-862e-4bc9-8957-e430acf53352 10.15.1.254 20.0.1.0/24 54.1.1.1
˓→ 3.1.1.0/24 1471902577 INACTIVE
de5d9a98-862e-4bc9-8957-e430acf53352 10.15.1.254 20.0.1.0/24 50.1.1.1
˓→ 3.1.1.0/24 1471902578 INACTIVE
--------------------------------------------------------------------------------------
˓→----------------------------------------------

*A:vsc# tools vswitch <nsg-ip> command "nuage-nsg-ike-cli show traffic-selectors"

--------------------------------------------------------------------------------------
˓→-------------------

Local TS -> Remote TS gw_name tunnel_status priority

˓→ conn_use_status
--------------------------------------------------------------------------------------
˓→-------------------

20.0.0.0/24 -> 2.1.1.0/24 My_Ikev2_Gw_Conn_2 UP 1002

˓→ INACTIVE
My_Ikev2_Gw_Conn_1 UP 1001
˓→ACTIVE

20.0.1.0/24 -> 3.1.1.0/24 My_Ikev2_Gw_Conn_3 UP 1003

˓→ ACTIVE

20.0.1.0/24 -> 3.2.1.0/24 My_Ikev2_Gw_Conn_4 UP 1004

˓→ ACTIVE

--------------------------------------------------------------------------------------
˓→-------------------

*A:vsc# tools vswitch <nsg-ip> command "nuage-nsg-ike-cli count enc-profile"

Number of configured encryption profiles: 2

--------------------------------------------------------------------------------------
˓→--------------------------

*A:vsc# tools vswitch <nsg-ip> command "nuage-nsg-ike-cli count gw-connections"

7.2. IKE IPsec 443

 VNS User Guide, Release 5.3.2

Number of configured gateway connections: 4

--------------------------------------------------------------------------------------
˓→--------------------------

*A:vsc# tools vswitch <nsg-ip> command "nuage-nsg-ike-cli count ipsec-tunnels"

Number of up tunnels: 4
Number of down tunnels: 0

--------------------------------------------------------------------------------------
˓→--------------------------

7.2. IKE IPsec 444

 CHAPTER

EIGHT

APPLICATION-AWARE ROUTING

• AAR (page 446)

• AAR Visualization (page 505)
• SaaS Routing (page 561)

445
 VNS User Guide, Release 5.3.2

8.1 AAR

• Overview (page 447)

– Application Discovery (AD) (page 447)
– Network Performance Measurement (NPM) (page 447)
– Application Policy and Visualization (APV) (page 447)
• Configuration Objects (page 448)
– Applications (page 448)
– APM Groups (page 448)
– Performance Monitors (page 449)
– Performance Monitor Scopes (page 449)
– NPM Groups (page 449)
• SLA-Aware Path Selection (page 449)
• AAR for NSG-UBR (page 451)
• Feature Details (page 451)
• Configuration Details (page 452)
– Enable Performance Management (page 452)
– Enable AD and APV (page 453)
– Define Applications (page 454)

* Define Performance Monitor Scope (page 456)

– Define Performance Monitor (page 458)
– Create APM Group (page 458)
– Create Application Binding (page 459)
– Associate APM Group with Domain (page 460)
– Define NPM Groups (page 461)

* Define Performance Monitor Scope (page 462)

• Provisioning Workflows (page 464)
– AD and APV Workflow (page 464)
– NPM Workflow (page 465)
• CLI Output (page 465)
• Performance Monitoring Debug CLI (page 468)
• DPI Debug CLI (page 503)

8.1. AAR 446

 VNS User Guide, Release 5.3.2

8.1.1 Overview

Application-Aware Routing (AAR) in VNS enables policy-driven application performance management in a multi-
path network. Specifically, AAR enables capabilities such as application discovery, network performance measure-
ment, and intelligent path selection for application traffic based on path performance. The primary business drivers are
the growth of hybrid WANs, use of the Internet as a viable transport option, and evolution of the WAN as an enabler
of cloud-based applications.
Traffic today is a mix of voice/video, bulk data transfer, web conferencing, ERP/CRM, SaaS etc.. Thus, businesses
want to measure and monitor network performance, identify applications traversing the WAN, and prioritize applica-
tion traffic such that business critical traffic can be routed over the best path with desired Service Level Agreement
(SLA) attributes, e.g. low latency. Certain application traffic, such as from social media sites may be dropped or routed
over a best-effort path.
The following are the functional components and building blocks of application-aware routing capabilities in VNS:

Application Discovery (AD)

Identify and classify network traffic coming into the access ports of a NSG on a per-application basis using:
• Signature-based L7 classification (e.g. Skype, Facebook, Google, etc) using a library of 1900+ signatures.
L7 classification also includes applications discovered using the Common Name in TLS certificates (when
available).
• Custom classification based on source/destination IP address, source/destination L4 ports, L4 Protocol
(TCP/UDP)
Use Case: User wants to monitor the traffic coming into the access ports of a NSG to understand the L7 profile (Skype,
Facebook, Google, etc) of traffic passing through their network.

Network Performance Measurement (NPM)

Measure and report health metrics (one way packet loss, jitter and latency) of overlay network connections between
NSGs in a domain using performance monitors, with a specified network profile (service class, payload size, traffic
rate), to measure path performance.
Use Case: User wants to monitor the health of the network connection between NSGs belonging to a particular
Domain. The user is interested in creating test traffic with a specific network profile (service class, payload size, traffic
rate) and monitor the one way packet loss, jitter and latency between the uplinks of different NSGs.

Application Policy and Visualization (APV)

Visualization is a key component of AAR and applies to AD and NPM as well. In this release, APV refers to policy-
based intelligent path selection for application traffic based on performance of the uplink. In the future, the scope of
APV will include broader policies for application-aware performance management.
Thus, currently APV enables application flows to be switched to another path, if the performance (One-way Latency,
Jitter, and Packet Loss) of an uplink degrades beyond the thresholds specified in the Application’s policy. Without
APV, the choice of the uplink for a given traffic flow is static.

Note: Traffic will only be performance routed if there are no advanced ACLs or if entries in the advanced ACLs have
the Local Uplink Preference set to Default.

8.1. AAR 447

 VNS User Guide, Release 5.3.2

Use Case: User wants to ensure that a particular Application is monitored for performance and if the performance of
the Application degrades below a certain threshold SLA (denoted by one way Latency, Jitter and Packet Loss) all the
flows belonging to the Application are switched over to a path that does not violate SLA, if such a path exists.

8.1.2 Configuration Objects

The following objects are utilized in the workflows for setting up AD, NPM, and APV. The detailed configuration
procedures are described later in this chapter.

Fig. 8.1: AAR Configuration Objects

Applications

An Application is used to classify traffic originating from access ports based on L4 and L7 characteristics. The
user can specify source/destination IP address, source/destination L4 ports, L4 Protocol (TCP/UDP) as well as a L7
classification. One or more flows may be mapped to an Application based on these characteristics.
An Application Service Level Agreement (SLA) denotes the maximum one-way Latency, Jitter and Packet Loss that is
acceptable for that Application. SLA is required for APV, and the application’s SLA metrics are compared against the
average latency, jitter and packet loss measurements reported for the uplinks. If the average values are greater than the
SLA values, then the system tries to select an alternate path that will satisfy the SLA parameters of that applications.
All flows beloging to the applications will be moved.

APM Groups

An APM (Application Performance Management) Group denotes a class of Applications that have similar network
characteristics, such as video, voice, bulk transfer, best-effort etc.. The APM Group is associated with a Performance
Monitor (page 449) which specifies the characteristics of the test traffic that is generated for measuring uplink perfor-
mance.
The test traffic is started for each APM group, instead of per application. Thus by grouping applications with similar
SLA requirements, the test traffic can be utilized efficiently.
An APM Group can:

8.1. AAR 448

 VNS User Guide, Release 5.3.2

• contain one or more Applications in a prioritized list (configurable by the user). Thus, if there are multiple
application definitions, then the priority will be leveraged to drive the match order.
• be mapped to one or more Domains with potentially a different priority per Domain.
The Applications belonging to a Domain will be flattened into an ordered list from high priority to low priority. This
ordering will take into account the Application Group priority and the Application Priority. A flow is matched to an
Application in the order of the prioritized list.

Performance Monitors

Performance Monitor denotes the characteristics of network traffic such as payload size, service class and traffic rate.
It is used to initiate the test traffic (also referred to as PM probe) for measuring the performance of the uplinks for
NPM and APV capabilities.

Note: NTP is used in the performance monitor infrastructure, and NTP drift errors may lead to false negatives in SLA
violations, especially in low latency networks.

For NPM, test traffic is executed continuously between all NSGs that are in scope. Link performance results are
reported by the receiving NSGs to VSD every 5 seconds.
For APV, test traffic is triggered via 1st packet detection (default) or is continuous or a combination of both.
1st packet detection means that once a flow is classified to given Application, it will trigger test traffic between the
source and destination NSG of the flow. Since Application classification may require a number of packets to be seen
by the classification engine, the triggering of the performance probe is delayed in contrast to when a continuous PM
probe is utilized. 1st packet probes are unicast probes (between two NSGs) and are optimized for CPU and network
utilization.
A trigger may be a combination of 1st packet detection and continuous. In this option, a continuous PM probe is
started between all NSGs in scope (as specified for an application). If a particular NSG is not in the list, then the probe
will be started when the 1st flow matching the Application is detected at that NSG.

Note: For APV related probes, there is an idle timeout of 150 seconds after which the probe session is terminated.

Performance Monitor Scopes

The Performance Monitor Scope object is used to define the list of source NSGs and destination NSGs whose network
paths need to be monitored using test traffic. The Performance Monitor Scopes may be associated with Application or
NPM Group.

NPM Groups

NPM (Network Performance Measurement) Group object is used for the NPM feature. It specifies the Performance
Monitor that will be used as a continuous probe between the NSGs defined by the Preformance Monitor Scope.

8.1.3 SLA-Aware Path Selection

You can create an advanced ACL that dynamically selects application flow paths based on a combination of local or
remote uplink preference and AAR default path selection behavior.

8.1. AAR 449

 VNS User Guide, Release 5.3.2

You can assign a specific application — or all applications in the domain — to an ingress forwarding policy entry.
When you specify an application, most other match criteria in the policy entry is disabled. This is because that criteria
are pulled directly from the selected application, and does not need to be configured on the policy entry. You can still
configure an origin and destination network.
If an application is selected as a match criterion, you can enable the SLA Aware parameter as an action. If the
parameter is enabled, paths are selected based on the advanced ACL uplink preference as well as SLA violation
information from the AAR analyzer. If the parameter is disabled, the advanced ACL uplink preference overrides AAR
path selection logic. You can configure either the Local Uplink Preference or Remote Uplink Preference parameter.
Specifying a remote uplink preference allows you to steer traffic from an NSG or NSG-UBR vPort to a specific network
port on the destination NSG. You cannot specifiy both a local and a remote uplink preference. One of the parameters
must remain Default. Remote uplink preference configuration allows for a use case where an NSG-UBR is forwarding
traffic to a destination NSG. The local uplink preference cannot be specified for an NSG-UBR.
When the SLA Aware parameter is enabled, the path selection preference order for each uplink preference is as
follows:
• Default: n/a
• Secondary: Sec. in SLA > Sec. out of SLA
• Primary: Prim. in SLA > Prim. out of SLA
• Prim:Sec: Prim. in SLA > Sec. in SLA > Prim. out of SLA > Sec. out of SLA
• Sec:Prim: Sec. in SLA > Prim. in SLA > Sec. out of SLA > Prim. out of SLA
• Symmetric: Prim./Prim. in SLA > Sec./Sec. in SLA > Prim./Sec. in SLA > Sec./Prim. in SLA > Prim./Prim.
out of SLA > Sec./Sec. out of SLA > Prim./Sec. out of SLA > Sec./Prim. out of SLA

Note: Refer to the uplink preference behavior table (page 211) to see the path selection behavior for advanced ACLs
that are not SLA-aware.

Note: Symmetric uplink preference is not supported for the Remote Uplink Preference.

Consider the following for path selection:

• When a path falls out of SLA, the path preference is dynamically reevaluated.
• If a higher preference path that is out of SLA becomes in SLA, the path selection reverts to the higher preference.
• In cases where the primary selection is actually multiple paths, the flow may revert to a secondary selection with
better performance.
• In cases where there are multiple equal-cost paths, the path with the best SLA for the Optimize For parameter
is selected.
• In cases where all paths are out of SLA, then the path with the best SLA for the Optimize For attribute is
selected.
If a path is selected past on pre-classification, the path selection logic will be overridden by the SLA-aware advanced
ACL. This can result in the traffic being switched to a new path.

Note: It is recommended that L3/L4 advanced ACL uplink preference be used instead of pre-classification.

8.1. AAR 450

 VNS User Guide, Release 5.3.2

Note: For L7 ACLs, IPv6 is not supported.

See the VSP User Guide for more information and workflows for configuring advanced ACLs in the VSD.

8.1.4 AAR for NSG-UBR

Traffic flowing between two NSG-UBRs is managed by UBR-AAR. UBR-AAR is agnostic of applications and peri-
odically steers traffic to the better path between two NSG-UBRs.
If an NSG-UBR has a vPort with DPI enabled, the traffic is managed by regular AAR. This means that NPM, APM,
and path switching is supported for traffic originating from an access-side NSG-UBR vPort towards a destination
access port on an NSG. This functionality allows the NSG-UBR to act as a transit in a domain if it is the only route
for AAR traffic flows.

8.1.5 Feature Details

Application Discovery: The ability to enable classification on Domain/Zone/Subnet/vPort level for the purpose of
discovering which applications are traversing a domain. No probing is required for this use case and as such no
intelligent path selection is done. Application Discovery process includes:
• Mirror all traffic ingressing on a vPort to the classifier (DPI engine). This is a vPort level attribute that is
inherited or locally configured.
• Identify and classify traffic into applications based on L4/L7/L4+L7 attribute match, independent of traffic
destination (overlay or underlay)
• Export results to statistics engine
• Generate reports for visualization of discovered applications

Note: When DPI is enabled, IPv4 traffic is mirrored. However, ARP packets are not mirrored and IPv6 is not
supported. As a result, the total measured application usage may not be equal to the port utilization. Additional
control traffic and statistics reporting may not be captured in the DPI, and tunnel encapsulation overhead may cause
the port utilization to be different from what is reflected in the AAR statistics even if all V-Ports are part of a single
domain with default application discovery.

Network Performance Measurement: The ability to turn on continuous probes between all NSGs for a given domain
for the purpose of measuring One-Way loss, delay, jitter. NPM process includes:
• Initiate probes independent of traffic ingressing the NSGs
• Calculate probe results by analyzing the response to received probe packets
• Export results to statistics engine
• Generate reports for visualization of performance measurement results
Application Policy and Visualization – Combination of the above two capabilities with the additional definition of
per application SLAs. In a multi-path network, each application will have its own best path from source to destina-
tion based on the traffic characteristics of the application. Using performance monitor probes to measure network
performance, an optimal path is dynamically selected to maintain application flows within a service-level contract.
Once Application SLAs are violated for a given path, all flows belonging to Application are moved to an alternate
conforming path. APV process includes:

8.1. AAR 451

 VNS User Guide, Release 5.3.2

• Trigger probes per application group for calculating path performance. Probes may have 1st packet trigger only
(in contrast to NPM, whereby probes must be set to continuous)
• Enforce VSD policies for applications with SLA definition by using path performance results to ensure that the
path does not violate the SLAs or select an alternate SLA conforming path
• Export SLA and application metrics to statistics engine
• Generate reports for visualization of Application flow information, byte/packet counts and SLA status

Note: When a TCP SYN packet is received within three minutes of a previous TCP SYN packet with the same
5-tuple, it is considered to be part of the same traffic flow. If the packet is received after the three minute window, it is
considered to be part of a new traffic flow which is mirrored for DPI.

8.1.6 Configuration Details

Enable Performance Management

AAR capabilities (AD, NPM, APV) must be enabled or disabled by the CSPRoot.
1. As CSPRoot, navigate to Organization Policies -> Organization Profiles, to create a new or modify an existing
profile.

8.1. AAR 452

 VNS User Guide, Release 5.3.2

Fig. 8.2: Enable AAR in Organization Profile

2. Check the Enable Performance Management option.

Note:
• Unless this option is enabled, AAR will not be available to the enterprises associated with this profile.
• The flag may be enabled or disabled at any time.

Enable AD and APV

The Deep Packet Inspection (DPI) flag enables the AD and APV capabilities.
1. As Enterprise Administrator, navigate to desired Domain/Zone/Subnet/vPort where AD or APV is desired.

8.1. AAR 453

 VNS User Guide, Release 5.3.2

2. On the far right, select icon, and scroll down in the rightmost panel until the DPI parameter is visible.

Fig. 8.3: Enable AD and APV capabilities

3. Select Enabled for invoking AD and APV capabilities.

Define Applications

Required for AD and APV use cases.

1. As Enterprise administrator, navigate to Applications tab -> Applications . Click on in the panel to add
a new Application or edit an existing one.

Note: A default Application is provided which cannot be edited. The default application matches any application.

2. In the popup enter the following attributes:

8.1. AAR 454

 VNS User Guide, Release 5.3.2

Fig. 8.4: Application Definition

• Name and Description

• Application match criteria using a known Application Signature or Custom match criteria based on:
– Protocol
– Source Port
– Destination Port
– From/To IP Match
– To/From IP Match
– Ether Type
– DSCP Marker
• Choose rule for Pre Classification Path to pin the local path selection. This preference will be used to steer
any flow that is matched using L4 attributes until the Application is potentially identified using L7 signature.
This is because signature-based match processing requires several packets for classification, and thus using
pre-classification, the Application can still be steered over a desired path.

Note: After L7 signature classification, path switching happens only if the path selected via pre-classification is out
of SLA.

The options are:

• None: Follow default preference, specifically Primary and if not available go over Secondary

8.1. AAR 455

 VNS User Guide, Release 5.3.2

• Primary: Follow Primary tagged uplinks only. If no Primary tagged uplinks are operational, drop traffic.
• Secondary: Follow Secondary tagged uplinks only. If no Secondary tagged uplinks are operational, drop traffic.
• Check Enable Performance Management for specifying the attributes required for APV. Enabling the flag
opens up the panel below it with the relevant attributes. The flag is optional:
– If not enabled, it allows the user to customize the reporting by just defining applications which will be
subsequently displayed in the reports
– If enabled, the defined applications can be routed based on path performance. In other words, enables APV
capability.
When enabled, the following additional attributes have to be specified:
• Select Optimize for criteria as either packet loss, delay or jitter. Thus, if there are multiple compliant paths,
then the path with the best SLA for the optimize for attribute is chosen. In addition, if all paths are in violation,
and there is no compliant path, then the path with the best SLA for optimize for attribute is chosen.
• Monitor Type
– First Packet (Application Triggered): Once traffic is classified to given Application at an NSG, it will
trigger unicast test traffic to the destination NSG.
– Continuous: If a particular NSG is in the Performance Monitor Scope list, it will start continuous probe to
its destination NSGs.
– First Packet and Continuous: If a particular NSG is in the Performance Monitor Scope list, it will start
continuous probe. If it is not in the list, then the probe will be started when the 1st flow matching the
Application is detected at that NSG.
• SLA attributes are used to measure compliance for application against performance monitor probe results. At
least one parameter value must be defined:
– Packet Loss: Define maximum average packet loss (one-way)
– Packet Delay: Define maximum average delay (one-way)
– Packet Jitter: Define maximum average jitter (one-way)

Note: Packet Loss calculation algorithm uses a fast fill procedure in which all probes will send out packets at a rate of
10 packets per second for the first 100 seconds until the buffer is filled up. After the fast fill is completed, the probes
will fall back to the configured packet rate. This improves the accuracy for slow rate probes.

Define Performance Monitor Scope

Required for the APV use case when continuous probes are utilized for monitoring path performance. As described
earlier, the probes may be triggered by 1st packet (post classification) or may be continuous probes. In the case of
continuous probe, a performance monitor scope is required so that the probe can be initiated between the source and
destination NSG pairs listed in the scope. Without the scope, the probes will not be initiated and as a result there will
be no path switching.

8.1. AAR 456

 VNS User Guide, Release 5.3.2

Fig. 8.5: Performance Monitor Scope Definition

1. For a selected Application, in the Performance Monitor Scopes panel, click on to define a new performance
monitor scope.
2. Enter a Name for the scope.
3. Select the list of Source and Destination NSGs in a domain whose network paths need to be monitored using
test traffic.

Note: For continuous performance monitoring probe, a performance monitor scope is required because it defines the
source and destination NSGs for the test traffic.

8.1. AAR 457

 VNS User Guide, Release 5.3.2

Define Performance Monitor

Required for NPM and APV use cases. Performance Monitor denotes the characteristics of network traffic such as
payload size, service class and traffic rate, and is used to initiate the test traffic with the specified profile.

1. As Enterprise administrator, navigate to Applications tab -> Performance Monitor . Click on in the panel
to add a new monitor or to edit an existing one.
2. Enter the following attributes:
• Name
• Description
• Service Class which is A-H. Default is service class H. Since the probe is intended to mimic the characteristics
of the traffic being monitored, ensure that the service class matches that set for Egress QoS (page 307).
• Payload Size in bytes. Minimum value is 91 Bytes.
• Probe Type to specify a one-way or HTTP probe. For AAR, select One-Way. See Reachability Monitoring
(page 562) for more information about configuring HTTP probes for IKE tunnels.
• Interval in packets per seconds. Rate of 10 packets per second (or 1 packet every 100 ms) is the maximum rate
at which a performance monitor probe can be sent.

Create APM Group

Required for AD and APV use cases. An APM Group denotes a class of Applications that have similar network
characteristics, such as video, voice, bulk transfer, best-effort etc..

1. As Enterprise administrator, navigate to Applications tab -> Application Performance Management Groups .
Click on in the panel to add a new monitor or to edit an existing one.

Note:
• A Default Application Group is provided for reporting discovered applications if no user defined applications
are configured. The default application group is associated with a Default Application.
• When Performance Management is enabled in the Organization profile, then the Default Application Group is
automatically assigned to domains in the Enterprise with a priority of 65535.

8.1. AAR 458

 VNS User Guide, Release 5.3.2

Fig. 8.6: New Application Performance Management Group

2. Enter Name and Description.

3. Associate a Performance Monitor with the APM Group. The PM is used to start the test traffic for measuring
the performance of the uplinks and monitoring SLA violations.

Note: If an APM Group has at least one application with Performance Management enabled, then a Performance
Monitor must be associated with the group.

Create Application Binding

Required for AD and APV use cases.

8.1. AAR 459

 VNS User Guide, Release 5.3.2

Fig. 8.7: New Application Binding

1. Once an APM Group is created, in the Application Binding panel, click on to add Applications to the
group with a certain priority. The flows will be matched against the Application in the prioritized order. Once
matched, the performance monitor probe associated with the corresponding APM Group (to which the applica-
tion belongs) will be picked for the test traffic.

Associate APM Group with Domain

Required for AD and APV use cases.

1. As Enterprise Administrator, navigate to Domain instance using the Networks tab.

2. On the far right panel, click on APM Group Binding . If an APM Group is already assigned, it will be listed
in the column below the icon.

8.1. AAR 460

 VNS User Guide, Release 5.3.2

Fig. 8.8: APM Group Binding

3. Click on to create the APM Group binding. In the popup, click on the icon which will display a list of
APM Groups available (for the enterprise) for consumption. Select an APM Group and assign a Priority to the
group.

Fig. 8.9: Associate APM Group to Domain

Attention: When associating APM Group to domain(s), ensure that the all the NSGs in performance monitor
scope are covered. Otherwise the performance monitor probe will not be initiated to an NSG that is not in the
domain.

Define NPM Groups

Required for the NPM use case.

NPM Group object is used for NPM feature and specifies the PM probe that will be used as a continuous probe between
the NSGs defined by the Performance Monitor Scope.

1. As Enterprise administrator, navigate to Applications tab -> NPM Groups . Click on in the panel to add
a new NPM object or to edit an existing one.

8.1. AAR 461

 VNS User Guide, Release 5.3.2

Fig. 8.10: Network Performance Measurement Definition

2. Enter Name and Description.

3. Associate a Performance Monitor with the NPM object. The Performance Monitor is used to start the test
traffic for network performance measurement continuous probe between the NSGs defined by the Performance
Monitor Scope.

Note:
• Performance Monitor association is mandatory for NPM.
• A default NPM object is provided with default Performance Monitor attributes and a default Performance Mon-
itor Scope (which is between all NSGs).

Fig. 8.11: Default NPM Group

Define Performance Monitor Scope

Required for the NPM use case.

8.1. AAR 462

 VNS User Guide, Release 5.3.2

Fig. 8.12: Performance Scope Definition

If NPM is desired between a certain set of NSGs, then the Performance Monitor Scope can be defined to limit the test
traffic.

1. For a selected NPM object, in the Performance Monitor Scopes panel, click on to define a new performance
monitor scope.
2. Enter a Name for the scope.
3. Select the list of Source and Destination NSGs in a domain whose network paths need to be monitored using
test traffic.

8.1. AAR 463

 VNS User Guide, Release 5.3.2

8.1.7 Provisioning Workflows

Each of the capabilities, AD, NPM, and APV invoke one or more configuration objects as summarized in the end-to-
end workflows presented in this section.
The workflows assume that the feature is enabled (page 452) by the CSPRoot.

AD and APV Workflow

Fig. 8.13: High-Level APV Provisioning Workflow

1. Enable DPI (page 453) on desired Domain/Zone/Subnet/vPort (right pane of respective object on VSD UI)
where AD/APV is required.
Since a Default Application Group is automatically associated with the domain with priority 65535, all Traffic will
be mapped to Default Application in the Default Application Group.

Note: The user can also create custom Applications (page 454) and associate them to APM Groups (page 459)
in a prioritized order. The APM Groups may be associated with a Domain (page 460) in a prioritized order. The
traffic flows will be matched againsts Applications in the APM Group and only if no match is found, will the Default
Application be chosen.

The following workflow represents a typical APV workflow.

2. Define Application (page 454) with L4/L7 match criteria and enable the Performance Management flag. Specify
associated SLAs, performance monitor, and other relevant performance routing related attributes.
3. Define Performance Monitor Scope (page 456).

8.1. AAR 464

 VNS User Guide, Release 5.3.2

4. Create APM Group (page 458) with the corresponding PM, and associate Applications (page 459).
5. Associate APM Group (page 460) to the domain with a priority specified. Based on the trigger selected, contin-
uous PM probe may be initiated or PM probe will start post classification. Uplink performance measurements
and SLA monitoring procedure will determine the best path for the application traffic.
6. SLA violation results per NSG/Application are displayed in the AAR visualizations.

NPM Workflow

Fig. 8.14: High-Level NPM Provisioning Workflow

1. Define NPM Group (page 461) with the Performance Monitor Scope.
2. This will start continuous Performance Monitor probes between all NSGs in scope.
3. Performance measurement results are displayed in the AAR visualizations.
4. Continuous PM probes can be stopped by de-associating the Performance Scope.

8.1.8 CLI Output

===============================================================================
*A:vsc# show vswitch-controller enterprise "test_organization" application detail
===============================================================================
Application Table
===============================================================================
App Name : Default Application
Id : 47554a11-b181-4b7a-a516-e6ba59b0da41
Sla Delay : 0 Sla Jitter : 0
Sla Loss : 0 Bandwidth : 0
Trigger : first-packet Symmetry : n/a
Opt Path Classify : noopt L7 Classify : *
Pre Path Classify : no-sel
Protocol : 0
Src Ip Range : *
Dest Ip Range : *
Src Port Range : *

8.1. AAR 465

 VNS User Guide, Release 5.3.2

Dest Port Range : *

Num Scopes : 0
App Name : port80
Id : ba4199f0-01f3-4a31-91af-eb4a8c3f1308
Sla Delay : 0 Sla Jitter : 0
Sla Loss : 1 Bandwidth : 0
Trigger : first-packet Symmetry : n/a
Opt Path Classify : noopt L7 Classify : *
Pre Path Classify : secondary
Protocol : 6
Src Ip Range : *
Dest Ip Range : *
Src Port Range : *
Dest Port Range : 80-80
Num Scopes : 0
===============================================================================

*A:vsc# show vswitch-controller enterprise "test_organization" application-group

˓→detail

===============================================================================
Application Group Table
===============================================================================
Name : Default Application Group
Id : 07346f1c-ee90-49a1-8804-b933d9d6ea9c
Probe : n/a
Num App Binds : 1 Num Domain Binds : 1
Domain : 824403975
-------------------------------------------------------------------------------
Application Group Application Binding Table
-------------------------------------------------------------------------------
App Name Priority
-------------------------------------------------------------------------------
Default Application 0
-------------------------------------------------------------------------------
No. of Application Bindings: 1
-------------------------------------------------------------------------------

Name : apm
Id : 9764d7f0-41a0-4aa8-be94-3c86de7ea9d1
Probe Id : 22db1f3c-8c7e-4188-b57d-ad8c01e42671
Probe Payload : 1400 Bytes Probe Rate : 1.0 pps
Probe Dscp : 0
Num App Binds : 1 Num Domain Binds : 1
Domain : 824403975
-------------------------------------------------------------------------------
Application Group Application Binding Table
-------------------------------------------------------------------------------
App Name Priority
-------------------------------------------------------------------------------
port80 10
-------------------------------------------------------------------------------
No. of Application Bindings: 1
-------------------------------------------------------------------------------

===============================================================================
*A:vsc# show vswitch-controller enterprise "test_organization" network-performance-
˓→management detail

8.1. AAR 466

 VNS User Guide, Release 5.3.2

===============================================================================
Network Performance Management
===============================================================================
Name : Default Network Performance Measurement
Id : 5fa3e921-2663-4047-a360-c0c458ef3219
Probe Id : 1816c43b-c8f0-456e-ad52-fc49dd92b332
Probe Payload : 576 Bytes Probe Rate : 0.100 pps
Probe Dscp : 0
Domain : 824403975
-------------------------------------------------------------------------------
Monitor Scopes
-------------------------------------------------------------------------------
Name : Default scope attached with default NPM
Id : f8587dc0-6efe-4100-8807-d80b4c359bff
Sources : *
Destinations : *

-------------------------------------------------------------------------------

===============================================================================

*A:vsc# show vswitch-controller enterprise "test_organization" domain "ncpeDomain1"

˓→detail

===============================================================================
Domain Table
===============================================================================
Domain name : ncpeDomain1
Enterprise name : test_organization
No. of vports : 16 VNID : 1045751
# of subnets : 4 Tunnel Type : VXLAN
BH Svc Id : 270600487
DHCP Behavior : consume
DHCP Relay Server : n/a
Max. ECMP Nhops : 8
GRT Leak Enabled : False Is Leakable Svc : False
Num App-Grp Binds : 2

-------------------------------------------------------------------------------
Uplink Route Distinguishers
-------------------------------------------------------------------------------
Type Route Distinguisher
-------------------------------------------------------------------------------
unknown value (18094) 65534:40614
unknown value (61887) 65534:18094
unknown value (43468) 65534:61887
disabled 65534:43468
-------------------------------------------------------------------------------
No. of Uplink Rd's: 4
-------------------------------------------------------------------------------

-------------------------------------------------------------------------------
Domain Application Group Bindings
-------------------------------------------------------------------------------
Application Group Priority
-------------------------------------------------------------------------------
apm 5

8.1. AAR 467

 VNS User Guide, Release 5.3.2

Default Application Group 65535

-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
Total No. of Domains: 1
===============================================================================

8.1.9 Performance Monitoring Debug CLI

The commands listed in this section help verify neighbor NSG information and probe information.
These OVS commands can also be run on the VSC using the tools vswitch command, specifying the IP address
of the OVS on which the command is to be executed.
Example:
*VSC# tools vswitch 192.15.1.254 command "ovs-appctl -t nuage_perfd
neighbor/dump"
Generator commands

ovs-appctl -t nuage_perfd generator/count

ovs-appctl -t nuage_perfd generator/dump
ovs-appctl -t nuage_perfd generator/dump type=NPM
ovs-appctl -t nuage_perfd generator/dump type=APM
ovs-appctl -t nuage_perfd generator/dump type=UBR
ovs-appctl -t nuage_perfd generator/dump type=NPM subtype=NONE
ovs-appctl -t nuage_perfd generator/dump type=NPM subtype=NAT_IPSEC
ovs-appctl -t nuage_perfd generator/dump type=NPM subtype=NAT_VXLAN
ovs-appctl -t nuage_perfd generator/dump json
ovs-appctl -t nuage_perfd generator/dump <xmpg uuid>
ovs-appctl -t nuage_perfd generator/dump <xmpg uuid> <details>
ovs-appctl -t nuage_perfd generator/dump <dpid>
ovs-appctl -t nuage_perfd generator/dump <dpid> <details>
ovs-appctl -t nuage_perfd generator/dump <xmpg uuid> <dpid> <details>

Analyzer commands

ovs-appctl -t nuage_perfd probestats/dump

ovs-appctl -t nuage_perfd analyzer/count
ovs-appctl -t nuage_perfd analyzer/dump
ovs-appctl -t nuage_perfd analyzer/dump json
ovs-appctl -t nuage_perfd analyzer/dump type=NPM
ovs-appctl -t nuage_perfd analyzer/dump type=APM
ovs-appctl -t nuage_perfd analyzer/dump type=UBR
ovs-appctl -t nuage_perfd analyzer/dump type=NPM subtype=NONE
ovs-appctl -t nuage_perfd analyzer/dump type=NPM subtype=NAT_IPSEC
ovs-appctl -t nuage_perfd analyzer/dump type=NPM subtype=NAT_VXLAN
ovs-appctl -t nuage_perfd analyzer/dump <xpmg uuid>
ovs-appctl -t nuage_perfd analyzer/dump <xpmg uuid> <details>
ovs-appctl -t nuage_perfd analyzer/dump <dpid>
ovs-appctl -t nuage_perfd analyzer/dump <dpid> <details>
ovs-appctl -t nuage_perfd analyzer/dump <xmpg uuid> <dpid> <details>

Neighbor commands

ovs-appctl -t nuage_perfd local/dump

ovs-appctl -t nuage_perfd local/dump json

8.1. AAR 468

 VNS User Guide, Release 5.3.2

ovs-appctl -t nuage_perfd neighbor/dump

ovs-appctl -t nuage_perfd neighbor/dump json

XPMG commands

ovs-appctl -t nuage_perfd xpmg/dump

ovs-appctl -t nuage_perfd xpmg/dump --npmg
ovs-appctl -t nuage_perfd xpmg/dump --apmg
ovs-appctl -t nuage_perfd xpmg/dump --ubrg
ovs-appctl -t nuage_perfd xpmg/dump --npmg details
ovs-appctl -t nuage_perfd xpmg/dump --apmg details
ovs-appctl -t nuage_perfd xpmg/dump --ubrg details
ovs-appctl -t nuage_perfd xpmg/dump json
ovs-appctl -t nuage_perfd probe_config/dump

Log commands

ovs-appctl -t nuage_perfd event-log/flush

ovs-appctl -t nuage_perfd event-log/dump
ovs-appctl -t nuage_perfd vlog/list | grep aar

Application commands

ovs-appctl -t nuage_perfd app/dump

ovs-appctl -t nuage_perfd app/dump <apmg uuid>
ovs-appctl -t nuage_perfd app/dump json

SLA violation commands

ovs-appctl -t nuage_perfd slaviolations/dump

ovs-appctl -t nuage_perfd slaviolations/dump <xpmg uuid> <dpid>
ovs-appctl -t nuage_perfd slaviolations/dump json

NSG-UBR commands

ovs-appctl -t nuage_perfd ubr/dump

ovs-appctl -t nuage_perfd ubr_ovs_published/dump
ovs-appctl -t nuage_perfd ubr/dump json
ovs-appctl -t nuage_perfd nsg/dump
ovs-appctl -t nuage_perfd nsg_ovs_published/dump
ovs-appctl -t nuage_perfd nsg/dump json

Generator command samples

The ovs-appctl -t nuage_perfd generator/count command displays the counters for path states of
all generators.
INIT: Path is in init state.
START_SENT: Control session is being established on this path.
RUNNING: Probes are being sent on this path.
STOP_SENT: Session teardown in progress on this path.
RST_RCVD: Session is being re-established on this path.
STOPPED: Session has been terminated.
UNKNOWN: Path is in error condition.

8.1. AAR 469

 VNS User Guide, Release 5.3.2

[root@nsg-73-106-59-247 nuage]# ovs-appctl -t nuage_perfd generator/count

Path Count
INIT : 24
START_SENT : 0
RUNNING : 24
STOP_SENT : 0
RST_RCVD : 0
STOPPED : 0
UNKNOWN : 0

The ovs-appctl -t nuage_perfd generator/dump command displays a summary of all the generators
and their generator states.
CREATED: Generator is created but is not active.
INITIALIZING: Control session is being established on some or all valid paths.
RUNNING: Probes are running on all valid paths.
STOPPED: Control session is terminated on all valid paths.
The following filters are supported for this command:
type=[NPM|APM|UBR]
subtype=[NONE|NAT_IPSEC|NAT_VXLAN]
ovs-appctl -t nuage_perfd generator/dump type=NPM
ovs-appctl -t nuage_perfd generator/dump type=APM
ovs-appctl -t nuage_perfd generator/dump type=UBR
ovs-appctl -t nuage_perfd generator/dump type=NPM subtype=NONE
ovs-appctl -t nuage_perfd generator/dump type=NPM subtype=NAT_IPSEC
ovs-appctl -t nuage_perfd generator/dump type=NPM subtype=NAT_VXLAN

<xpmg> [details]
<dpid> [details]
<xpmg> <dpid> <details>
ovs-appctl -t nuage_perfd generator/dump <xmpg uuid>
ovs-appctl -t nuage_perfd generator/dump <dpid>
ovs-appctl -t nuage_perfd generator/dump <xmpg uuid> details
ovs-appctl -t nuage_perfd generator/dump <dpid> details
ovs-appctl -t nuage_perfd generator/dump <xmpg uuid> <dpid> details

ovs-appctl -t nuage_perfd generator/dump uuid=<xpmg uuid>

ovs-appctl -t nuage_perfd generator/dump uuid=<xpmg uuid> details
ovs-appctl -t nuage_perfd generator/dump uuid=<xpmg uuid> dpid=<dpid> details

ovs-appctl -t nuage_perfd generator/dump json

[root@nsg-73-106-59-247 nuage]# ovs-appctl -t nuage_perfd generator/dump

UUID IP
˓→GENERATOR_STATE PROBE_RUNNING PROBE_STOPPED PROBE_STARTING
˓→ PROBE_STOPPING PATH_VALID

5b311c55-ff9b-4c42-b83b-eb7e5d03fb4a 2.231.64.98
˓→ RUNNING 4 0 0
˓→ 0 4

8.1. AAR 470

 VNS User Guide, Release 5.3.2

4908104c-1749-4601-8845-daab95eb810f 103.126.194.189
˓→ RUNNING 4 0 0
˓→ 0 4

4908104c-1749-4601-8845-daab95eb810f 2.231.64.98
˓→ RUNNING 4 0 0
˓→ 0 4

4908104c-1749-4601-8845-daab95eb810f 70.127.142.214
˓→ RUNNING 4 0 0
˓→ 0 4

4eaa6c3c-7de5-42fa-9ba9-7943f4e95313 2.231.64.98
˓→ CREATED 0 0 0
˓→ 0 0

37d34d4d-b950-49d9-8509-51d937e23ef1 2.231.64.98
˓→ CREATED 0 0 0
˓→ 0 0

5b311c55-ff9b-4c42-b83b-eb7e5d03fb4a 70.127.142.214
˓→ RUNNING 4 0 0
˓→ 0 4

4eaa6c3c-7de5-42fa-9ba9-7943f4e95313 70.127.142.214
˓→ CREATED 0 0 0
˓→ 0 0

37d34d4d-b950-49d9-8509-51d937e23ef1 70.127.142.214
˓→ CREATED 0 0 0
˓→ 0 0

4eaa6c3c-7de5-42fa-9ba9-7943f4e95313 103.126.194.189
˓→ CREATED 0 0 0
˓→ 0 0

5b311c55-ff9b-4c42-b83b-eb7e5d03fb4a 103.126.194.189
˓→ RUNNING 4 0 0
˓→ 0 4

37d34d4d-b950-49d9-8509-51d937e23ef1 103.126.194.189
˓→ CREATED 0 0 0
˓→ 0 0

[root@nsg-73-106-59-247 nuage]# ovs-appctl -t nuage_perfd generator/dump type=NPM

UUID IP
˓→GENERATOR_STATE PROBE_RUNNING PROBE_STOPPED PROBE_STARTING
˓→ PROBE_STOPPING PATH_VALID

5b311c55-ff9b-4c42-b83b-eb7e5d03fb4a 2.231.64.98
˓→ RUNNING 4 0 0
˓→ 0 4

4eaa6c3c-7de5-42fa-9ba9-7943f4e95313 2.231.64.98
˓→ CREATED 0 0 0
˓→ 0 0

8.1. AAR 471

 VNS User Guide, Release 5.3.2

37d34d4d-b950-49d9-8509-51d937e23ef1 2.231.64.98
˓→ CREATED 0 0 0
˓→ 0 0

5b311c55-ff9b-4c42-b83b-eb7e5d03fb4a 70.127.142.214
˓→ RUNNING 4 0 0
˓→ 0 4

4eaa6c3c-7de5-42fa-9ba9-7943f4e95313 70.127.142.214
˓→ CREATED 0 0 0
˓→ 0 0

37d34d4d-b950-49d9-8509-51d937e23ef1 70.127.142.214
˓→ CREATED 0 0 0
˓→ 0 0

4eaa6c3c-7de5-42fa-9ba9-7943f4e95313 103.126.194.189
˓→ CREATED 0 0 0
˓→ 0 0

5b311c55-ff9b-4c42-b83b-eb7e5d03fb4a 103.126.194.189
˓→ RUNNING 4 0 0
˓→ 0 4

37d34d4d-b950-49d9-8509-51d937e23ef1 103.126.194.189
˓→ CREATED 0 0 0
˓→ 0 0

[root@nsg-73-106-59-247 nuage]# ovs-appctl -t nuage_perfd generator/dump type=APM

UUID IP
˓→GENERATOR_STATE PROBE_RUNNING PROBE_STOPPED PROBE_STARTING
˓→ PROBE_STOPPING PATH_VALID

4908104c-1749-4601-8845-daab95eb810f 103.126.194.189
˓→ RUNNING 4 0 0
˓→ 0 4

4908104c-1749-4601-8845-daab95eb810f 2.231.64.98
˓→ RUNNING 4 0 0
˓→ 0 4

4908104c-1749-4601-8845-daab95eb810f 70.127.142.214
˓→ RUNNING 4 0 0
˓→ 0 4

[root@nsg-73-106-59-247 nuage]# ovs-appctl -t nuage_perfd generator/dump type=NPM

˓→subtype=NONE

UUID IP
˓→GENERATOR_STATE PROBE_RUNNING PROBE_STOPPED PROBE_STARTING
˓→ PROBE_STOPPING PATH_VALID

5b311c55-ff9b-4c42-b83b-eb7e5d03fb4a 2.231.64.98
˓→ RUNNING 4 0 0
˓→ 0 4

5b311c55-ff9b-4c42-b83b-eb7e5d03fb4a 70.127.142.214
˓→ RUNNING 4 0 0
˓→ 0 4

8.1. AAR 472

 VNS User Guide, Release 5.3.2

5b311c55-ff9b-4c42-b83b-eb7e5d03fb4a 103.126.194.189
˓→ RUNNING 4 0 0
˓→ 0 4

[root@nsg-2-231-64-98 nuage]# ovs-appctl -t nuage_perfd generator/dump json |

˓→ python -m json.tool
[
{
"APM Trigger count": 1,
"APMG_Details": {
"APMG_UUID": "4908104c-1749-4601-8845-daab95eb810f",
"APPS": [
{
"APMG_UUID": "4908104c-1749-4601-8845-daab95eb810f",
"APM_Trigger": 1,
"APM_UUID": "4d237281-6e4e-4548-a9d3-d350b3aa9f98",
"Is_AAR_Enabled": "true",
"Jitter": -1,
"Latency": 50,
"Pending_Action": 0,
"Pkt_Loss": -1
}
],
"Probe_Config": {
"Dscp": 0,
"Mtu": 576,
"Rate": 0.100000001490116
}
},
"Active_Probe_Count": 0,
"DPID": "73.106.59.247",
"Generator_State": "RUNNING",
"Generator_UUID": "4908104c-1749-4601-8845-daab95eb810f",
"Group_Type": 2,
"Local_NSG_IP": "2.231.64.98",
"Local_Update_Seq_No": 2,
"Local_Uplink_Info": [
{
"Ip": "10.15.33.254",
"Is_Functional": "true",
"Is_UBR": "false",
"Priority": 1,
"Underlay_Id": 0,
"Uplink": 0
},
{
"Ip": "10.18.33.254",
"Is_Functional": "true",
"Is_UBR": "false",
"Priority": 4,
"Underlay_Id": 0,
"Uplink": 1
}
],
"Neighbor_IP": "73.106.59.247",
"Neighbor_Uplink_Info": [
{

8.1. AAR 473

 VNS User Guide, Release 5.3.2

"Ip": "10.15.3.254",
"Is_Functional": "true",
"Is_UBR": "false",
"Priority": 1,
"Underlay_Id": 0,
"Uplink": 0
},
{
"Ip": "10.18.3.254",
"Is_Functional": "true",
"Is_UBR": "false",
"Priority": 4,
"Underlay_Id": 0,
"Uplink": 1
}
],
"Probe_Config_Info": {
"Dscp": 0,
"Mtu": 576,
"Probe_UUID": "4908104c-1749-4601-8845-daab95eb810f",
"Rate": 0.100000001490116
},
"Probe_Info": [
{
"Current_packet_Sequence_No": 0,
"Last_packet_Sequence_No": 515,
"Local_Uplink_Index": 0,
"Path": 0,
"Remote_Uplink_Index": 0,
"Session_Id": 1,
"Start_Ack_Received_Timestamp": 1516896740329581327,
"Start_Retry": 0,
"Start_Sent_Timestamp": 1516896740186931577,
"State": "UP",
"Stop_Retry": 0
},
{
"Current_packet_Sequence_No": 0,
"Last_packet_Sequence_No": 515,
"Local_Uplink_Index": 0,
"Path": 1,
"Remote_Uplink_Index": 0,
"Session_Id": 1,
"Start_Ack_Received_Timestamp": 1516896740329581327,
"Start_Retry": 0,
"Start_Sent_Timestamp": 1516896740186931577,
"State": "UP",
"Stop_Retry": 0
},
{
"Current_packet_Sequence_No": 0,
"Last_packet_Sequence_No": 515,
"Local_Uplink_Index": 0,
"Path": 2,
"Remote_Uplink_Index": 0,
"Session_Id": 1,
"Start_Ack_Received_Timestamp": 1516896740329581327,
"Start_Retry": 0,

8.1. AAR 474

 VNS User Guide, Release 5.3.2

"Start_Sent_Timestamp": 1516896740186931577,
"State": "UP",
"Stop_Retry": 0
},
{
"Current_packet_Sequence_No": 0,
"Last_packet_Sequence_No": 515,
"Local_Uplink_Index": 0,
"Path": 3,
"Remote_Uplink_Index": 0,
"Session_Id": 1,
"Start_Ack_Received_Timestamp": 1516896740329581327,
"Start_Retry": 0,
"Start_Sent_Timestamp": 1516896740186931577,
"State": "UP",
"Stop_Retry": 0
}
],
"Sla_Config_Timer": {
"Expiry_Time": 25804818,
"Expiry_Time_Ticks": 129024090,
"Time_Interval": 20,
"Time_Interval_Ticks": 100,
"Time_Left": 0,
"Time_Left_Ticks": 0,
"Timer Active": "false"
},
"Tx_App_Config": [
{
"APP": {
"APMG_UUID": "4908104c-1749-4601-8845-daab95eb810f",
"APM_Trigger": 1,
"APM_UUID": "4d237281-6e4e-4548-a9d3-d350b3aa9f98",
"Is_AAR_Enabled": "true",
"Jitter": -1,
"Latency": 50,
"Pending_Action": 0,
"Pkt_Loss": -1
},
"Is_Loaded_By_Flow": "false",
"Is_Loaded_By_Scope": "true",
"Retry_Count": 0,
"Sla_Config_Operation": 0,
"Sla_Config_State": 2
}
],
"Tx_App_Violation_Cache": [],
"Tx_Path_Info": [
{
"Dst_Port": 50000,
"Is_UBR": "false",
"Path_State": 1,
"Src_Port": 50000
},
{
"Dst_Port": 50003,
"Is_UBR": "false",
"Path_State": 1,

8.1. AAR 475

 VNS User Guide, Release 5.3.2

"Src_Port": 50000
},
{
"Dst_Port": 50000,
"Is_UBR": "false",
"Path_State": 1,
"Src_Port": 50003
},
{
"Dst_Port": 50003,
"Is_UBR": "false",
"Path_State": 1,
"Src_Port": 50003
}
],
"Tx_Runtime_APP_Config": []
}
]

[root@nsg-73-106-59-247 nuage]# ovs-appctl -t nuage_perfd generator/dump a31e762c-

˓→1bc6-40cb-8219-5f96fdcce666

UUID IP
˓→GENERATOR_STATE PROBE_RUNNING PROBE_STOPPED PROBE_STARTING
˓→ PROBE_STOPPING PATH_VALID

a31e762c-1bc6-40cb-8219-5f96fdcce666 2.231.64.98
˓→ RUNNING 4 0 0
˓→ 0 4
[root@nsg-73-106-59-247 nuage]# ovs-appctl -t nuage_perfd generator/dump a31e762c-
˓→1bc6-40cb-8219-5f96fdcce666 details

Generator State: RUNNING

Path: 0, Path State: UP, Start Retry: 0, Stop Retry: 0, Probe State: Running, Session
˓→ID: 1, Last Sequence number: 908
Path: 1, Path State: UP, Start Retry: 0, Stop Retry: 0, Probe State: Running, Session
˓→ID: 1, Last Sequence number: 908
Path: 2, Path State: UP, Start Retry: 0, Stop Retry: 0, Probe State: Running, Session
˓→ID: 1, Last Sequence number: 908
Path: 3, Path State: UP, Start Retry: 0, Stop Retry: 0, Probe State: Running, Session
˓→ID: 1, Last Sequence number: 908

Grp Type: 1
Probe Config Data
PROBE_UUID: a31e762c-1bc6-40cb-8219-5f96fdcce666
Dscp : 0, Rate : 10.000000, Mtu : 500
Local NSG data
Dpid: 73.106.59.247
Uplink Ip: 10.15.3.254,
Priority: 1
Underlay Id: 0
Is UBR Uplink: false
Is Functional: true
Enable NAT Probes: false
Traffic Through UBR Only: false

Uplink Ip: 10.18.3.254,

Priority: 4
Underlay Id: 0

8.1. AAR 476

 VNS User Guide, Release 5.3.2

Is UBR Uplink: false

Is Functional: true
Enable NAT Probes: false
Traffic Through UBR Only: false

Neighbor NSG Data

Dpid: 2.231.64.98 Domain:
Uplink Ip: 10.15.33.254,
Priority: 1
Underlay Id: 0
Is UBR Uplink: false
Is Functional: true
Enable NAT Probes: true,
Traffic Through UBR Only: false

Uplink Ip: 10.18.33.254,

Priority: 4
Underlay Id: 0
Is UBR Uplink: false
Is Functional: true
Enable NAT Probes: true,
Traffic Through UBR Only: false

[root@nsg-73-106-59-247 nuage]# ovs-appctl -t nuage_perfd generator/dump 2.231.64.98

UUID IP
˓→GENERATOR_STATE PROBE_RUNNING PROBE_STOPPED PROBE_STARTING
˓→ PROBE_STOPPING PATH_VALID

a31e762c-1bc6-40cb-8219-5f96fdcce666 2.231.64.98
˓→ RUNNING 4 0 0
˓→ 0 4

[root@nsg-73-106-59-247 nuage]# ovs-appctl -t nuage_perfd generator/dump 2.231.64.98

˓→details

Generator State: RUNNING

Path: 0, Path State: UP, Start Retry: 0, Stop Retry: 0, Probe State: Running, Session
˓→ID: 1, Last Sequence number: 1158
Path: 1, Path State: UP, Start Retry: 0, Stop Retry: 0, Probe State: Running, Session
˓→ID: 1, Last Sequence number: 1158
Path: 2, Path State: UP, Start Retry: 0, Stop Retry: 0, Probe State: Running, Session
˓→ID: 1, Last Sequence number: 1158
Path: 3, Path State: UP, Start Retry: 0, Stop Retry: 0, Probe State: Running, Session
˓→ID: 1, Last Sequence number: 1158

Grp Type: 1
Probe Config Data
PROBE_UUID: a31e762c-1bc6-40cb-8219-5f96fdcce666
Dscp : 0, Rate : 10.000000, Mtu : 500
Local NSG data
Dpid: 73.106.59.247
Uplink Ip: 10.15.3.254,
Priority: 1
Underlay Id: 0
Is UBR Uplink: false
Is Functional: true
Enable NAT Probes: false

8.1. AAR 477

 VNS User Guide, Release 5.3.2

Traffic Through UBR Only: false

Uplink Ip: 10.18.3.254,

Priority: 4
Underlay Id: 0
Is UBR Uplink: false
Is Functional: true
Enable NAT Probes: false
Traffic Through UBR Only: false

Neighbor NSG Data

Dpid: 2.231.64.98 Domain:
Uplink Ip: 10.15.33.254,
Priority: 1
Underlay Id: 0
Is UBR Uplink: false
Is Functional: true
Enable NAT Probes: true,
Traffic Through UBR Only: false

Uplink Ip: 10.18.33.254,

Priority: 4
Underlay Id: 0
Is UBR Uplink: false
Is Functional: true
Enable NAT Probes: true,
Traffic Through UBR Only: false

[root@nsg-73-106-59-247 nuage]# ovs-appctl -t nuage_perfd generator/dump a31e762c-

˓→1bc6-40cb-8219-5f96fdcce666 2.231.64.98 details

Generator State: RUNNING

Path: 0, Path State: UP, Start Retry: 0, Stop Retry: 0, Probe State: Running, Session
˓→ID: 1, Last Sequence number: 1394
Path: 1, Path State: UP, Start Retry: 0, Stop Retry: 0, Probe State: Running, Session
˓→ID: 1, Last Sequence number: 1394
Path: 2, Path State: UP, Start Retry: 0, Stop Retry: 0, Probe State: Running, Session
˓→ID: 1, Last Sequence number: 1394
Path: 3, Path State: UP, Start Retry: 0, Stop Retry: 0, Probe State: Running, Session
˓→ID: 1, Last Sequence number: 1394

Grp Type: 1
Probe Config Data
PROBE_UUID: a31e762c-1bc6-40cb-8219-5f96fdcce666
Dscp : 0, Rate : 10.000000, Mtu : 500
Local NSG data
Dpid: 73.106.59.247
Uplink Ip: 10.15.3.254,
Priority: 1
Underlay Id: 0
Is UBR Uplink: false
Is Functional: true
Enable NAT Probes: false
Traffic Through UBR Only: false

Uplink Ip: 10.18.3.254,

Priority: 4
Underlay Id: 0

8.1. AAR 478

 VNS User Guide, Release 5.3.2

Is UBR Uplink: false

Is Functional: true
Enable NAT Probes: false
Traffic Through UBR Only: false

Neighbor NSG Data

Dpid: 2.231.64.98 Domain:
Uplink Ip: 10.15.33.254,
Priority: 1
Underlay Id: 0
Is UBR Uplink: false
Is Functional: true
Enable NAT Probes: true,
Traffic Through UBR Only: false

Uplink Ip: 10.18.33.254,

Priority: 4
Underlay Id: 0
Is UBR Uplink: false
Is Functional: true
Enable NAT Probes: true,
Traffic Through UBR Only: false

[root@nsg-73-106-59-247 nuage]# ovs-appctl -t nuage_perfd generator/dump type=APM

UUID IP
˓→GENERATOR_STATE PROBE_RUNNING PROBE_STOPPED PROBE_STARTING
˓→ PROBE_STOPPING PATH_VALID

4908104c-1749-4601-8845-daab95eb810f 103.126.194.189
˓→ RUNNING 4 0 0
˓→ 0 4

4908104c-1749-4601-8845-daab95eb810f 2.231.64.98
˓→ RUNNING 4 0 0
˓→ 0 4

4908104c-1749-4601-8845-daab95eb810f 70.127.142.214
˓→ CREATED 0 0 0
˓→ 0 0

[root@nsg-73-106-59-247 nuage]# ovs-appctl -t nuage_perfd generator/dump type=NPM

˓→subtype=NAT_IPSEC

UUID IP
˓→GENERATOR_STATE PROBE_RUNNING PROBE_STOPPED PROBE_STARTING
˓→ PROBE_STOPPING PATH_VALID

4eaa6c3c-7de5-42fa-9ba9-7943f4e95313 70.127.142.214
˓→ RUNNING 4 0 0
˓→ 0 4

4eaa6c3c-7de5-42fa-9ba9-7943f4e95313 103.126.194.189
˓→ RUNNING 4 0 0
˓→ 0 4

4eaa6c3c-7de5-42fa-9ba9-7943f4e95313 2.231.64.98
˓→ RUNNING 4 0 0
˓→ 0 4

8.1. AAR 479

 VNS User Guide, Release 5.3.2

[root@nsg-73-106-59-247 nuage]# ovs-appctl -t nuage_perfd generator/dump type=NPM

˓→subtype=NAT_VXLAN

UUID IP
˓→GENERATOR_STATE PROBE_RUNNING PROBE_STOPPED PROBE_STARTING
˓→ PROBE_STOPPING PATH_VALID

37d34d4d-b950-49d9-8509-51d937e23ef1 70.127.142.214
˓→ RUNNING 4 0 0
˓→ 0 4

37d34d4d-b950-49d9-8509-51d937e23ef1 2.231.64.98
˓→ RUNNING 4 0 0
˓→ 0 4

37d34d4d-b950-49d9-8509-51d937e23ef1 103.126.194.189
˓→ RUNNING 4 0 0
˓→ 0 4

Analyzer command samples

The ovs-appctl -t nuage_perfd probestats/dump command displays all analyzer instances at the path
level with current delay, jitter and packet loss values.

[root@nsg-2-231-64-98 nuage]# ovs-appctl -t nuage_perfd probestats/dump

XPMG UUID SRC DPID DST DPID SRC UPLINK DST
˓→UPLINK LATENCY JITTER LOSS
4908104c-1749-4601-8845-daab95eb810f 73.106.59.247 2.231.64.98 primary1
˓→primary1 3.275 0.585 0.000
4908104c-1749-4601-8845-daab95eb810f 73.106.59.247 2.231.64.98 primary1
˓→secondary2 2.801 0.712 0.000
4908104c-1749-4601-8845-daab95eb810f 73.106.59.247 2.231.64.98 secondary2
˓→primary1 2.771 0.528 0.000
4908104c-1749-4601-8845-daab95eb810f 73.106.59.247 2.231.64.98 secondary2
˓→secondary2 3.037 0.600 0.000
4eaa6c3c-7de5-42fa-9ba9-7943f4e95313 73.106.59.247 2.231.64.98 primary1
˓→primary1 2.937 0.616 0.000
4eaa6c3c-7de5-42fa-9ba9-7943f4e95313 73.106.59.247 2.231.64.98 primary1
˓→secondary2 2.863 0.513 0.000
4eaa6c3c-7de5-42fa-9ba9-7943f4e95313 73.106.59.247 2.231.64.98 secondary2
˓→primary1 2.935 0.714 0.000
4eaa6c3c-7de5-42fa-9ba9-7943f4e95313 73.106.59.247 2.231.64.98 secondary2
˓→secondary2 3.103 0.652 0.000
37d34d4d-b950-49d9-8509-51d937e23ef1 73.106.59.247 2.231.64.98 primary1
˓→primary1 3.875 0.353 0.000
37d34d4d-b950-49d9-8509-51d937e23ef1 73.106.59.247 2.231.64.98 primary1
˓→secondary2 3.260 0.435 0.000
37d34d4d-b950-49d9-8509-51d937e23ef1 73.106.59.247 2.231.64.98 secondary2
˓→primary1 3.087 0.627 0.000
37d34d4d-b950-49d9-8509-51d937e23ef1 73.106.59.247 2.231.64.98 secondary2
˓→secondary2 3.672 0.063 0.000

The ovs-appctl -t nuage_perfd analyzer/count command displays the count of analyzer instances as
well as counts for path states.
INIT: Analyzer path has been initialized, but the session is not yet established.
RUNNING: Probes are running on the path.
STOPPED: The probe session has been terminated on the path.

8.1. AAR 480

 VNS User Guide, Release 5.3.2

[root@nsg-2-231-64-98 nuage]# ovs-appctl -t nuage_perfd analyzer/count

NUM_ANALYZERS : 1
Path Count
INIT : 0
RUNNING : 4
STOPPED : 0

The ovs-appctl -t nuage_perfd analyzer/dump command displays a summary of all analyzers and
path state counters.
The following filters are supported for this command:
type=[NPM|APM|UBR]
subtype=[NONE|NAT_IPSEC|NAT_VXLAN]

ovs-appctl -t nuage_perfd analyzer/dump type=NPM

ovs-appctl -t nuage_perfd analyzer/dump type=APM
ovs-appctl -t nuage_perfd analyzer/dump type=UBR
ovs-appctl -t nuage_perfd analyzer/dump type=NPM subtype=NONE
ovs-appctl -t nuage_perfd analyzer/dump type=NPM subtype=NAT_IPSEC
ovs-appctl -t nuage_perfd analyzer/dump type=NPM subtype=NAT_VXLAN

<xpmg> [details]
<dpid> [details]
<xpmg> <dpid> <details>

ovs-appctl -t nuage_perfd analyzer/dump <xmpg uuid>

ovs-appctl -t nuage_perfd analyzer/dump <dpid>
ovs-appctl -t nuage_perfd analyzer/dump <xmpg uuid> details
ovs-appctl -t nuage_perfd analyzer/dump <dpid> details
ovs-appctl -t nuage_perfd analyzer/dump <xmpg uuid> <dpid> details

ovs-appctl -t nuage_perfd analyzer/dump uuid=<xpmg uuid>

ovs-appctl -t nuage_perfd analyzer/dump uuid=<xpmg uuid> details
ovs-appctl -t nuage_perfd analyzer/dump dpid=<dpid>
ovs-appctl -t nuage_perfd analyzer/dump dpid=<dpid> details
ovs-appctl -t nuage_perfd analyzer/dump uuid=<xmpg uuid> dpid=<dpid> details

ovs-appctl -t nuage_perfd analyzer/dump json

[root@nsg-2-231-64-98 nuage]# ovs-appctl -t nuage_perfd analyzer/dump

UUID DPID PATH_
˓→RUNNING PATH_STOPPED

a31e762c-1bc6-40cb-8219-5f96fdcce666 73.106.59.247
˓→ 4 0

[root@nsg-2-231-64-98 nuage]# ovs-appctl -t nuage_perfd analyzer/dump json | python -

˓→m json.tool

[
{
"Analyzer_UUID": "4908104c-1749-4601-8845-daab95eb810f",
"DPID": "73.106.59.247",
"Group_Type": 2,
"In_Sla_Seq": 8,

8.1. AAR 481

 VNS User Guide, Release 5.3.2

"Jitter_Info": [],
"Latency_Info": [
{
"App_UUIDs": [
{
"APP_UUID": "4d237281-6e4e-4548-a9d3-d350b3aa9f98"
}
],
"Latency": 50
}
],
"Out_Check_Seq": 32,
"Packet_Loss_Info": [
{
"App_UUIDs": [
{
"APP_UUID": "d9d6227d-da35-4097-bfa5-8bd56309dbe5"
}
],
"PktLoss": 10
},
{
"App_UUIDs": [
{
"APP_UUID": "4d237281-6e4e-4548-a9d3-d350b3aa9f98"
}
],
"PktLoss": 100
}
],
"Path_Info": [
{
"Jitter": 0.573734402656555,
"Jitter_Miss": 2,
"Latency": 2.01721143722534,
"Path": 0,
"Pkt_Loss": 0,
"Session_Id": 1,
"State": "Running"
},
{
"Jitter": 0.116666346788406,
"Jitter_Miss": 2,
"Latency": 1.72442948818207,
"Path": 1,
"Pkt_Loss": 0,
"Session_Id": 1,
"State": "Running"
},
{
"Jitter": 0.528412878513336,
"Jitter_Miss": 2,
"Latency": 1.88200044631958,
"Path": 2,
"Pkt_Loss": 0,
"Session_Id": 1,
"State": "Running"
},

8.1. AAR 482

 VNS User Guide, Release 5.3.2

{
"Jitter": 0.380102902650833,
"Jitter_Miss": 2,
"Latency": 2.27389359474182,
"Path": 3,
"Pkt_Loss": 0,
"Session_Id": 1,
"State": "Running"
}
],
"Reporting_Timer": {
"Expiry_Time": 12905088,
"Expiry_Time_Ticks": 129050880,
"Time_Interval": 300,
"Time_Interval_Ticks": 3000,
"Time_Left": 179,
"Time_Left_Ticks": 1790,
"Timer Active": "true"
},
"Rx_App_Slas": [
{
"App_Id": "d9d6227d-da35-4097-bfa5-8bd56309dbe5",
"Jitter": -1,
"Latency": -1,
"Optimize_For": 2,
"Packet_Loss": 10
},
{
"App_Id": "4d237281-6e4e-4548-a9d3-d350b3aa9f98",
"Jitter": -1,
"Latency": 50,
"Optimize_For": 0,
"Packet_Loss": 100
}
],
"Sla_Check_Interval": 30000,
"Sla_Violation_Retry_Interval": 0,
"Validation_Timer": {
"Expiry_Time": 12905088,
"Expiry_Time_Ticks": 129050880,
"Time_Interval": 300,
"Time_Interval_Ticks": 3000,
"Time_Left": 179,
"Time_Left_Ticks": 1790,
"Timer Active": "true"
}
}
]

[root@nsg-2-231-64-98 nuage]# ovs-appctl -t nuage_perfd analyzer/dump type=NPM

UUID DPID PATH_
˓→RUNNING PATH_STOPPED

a31e762c-1bc6-40cb-8219-5f96fdcce666 73.106.59.247
˓→ 4 0

8.1. AAR 483

 VNS User Guide, Release 5.3.2

[root@nsg-2-231-64-98 nuage]# ovs-appctl -t nuage_perfd analyzer/dump type=NPM

˓→subtype=NONE

UUID DPID PATH_

˓→RUNNING PATH_STOPPED

a31e762c-1bc6-40cb-8219-5f96fdcce666 73.106.59.247
˓→ 4 0

[root@nsg-2-231-64-98 nuage]# ovs-appctl -t nuage_perfd analyzer/dump a31e762c-1bc6-

˓→ 40cb-8219-5f96fdcce666 details
Path: 0, Current State: Running, Session ID: 1. Latency: 1.033157, Jitter 0.524125,
˓→Pkt Loss: 0.000000, Jitter Miss: 1.000000, Last Sequence Number:2172

Path: 1, Current State: Running, Session ID: 1. Latency: 0.490650, Jitter 0.073372,
˓→Pkt Loss: 0.000000, Jitter Miss: 1.000000, Last Sequence Number:2172

Path: 2, Current State: Running, Session ID: 1. Latency: 0.790880, Jitter 0.178514,
˓→Pkt Loss: 0.000000, Jitter Miss: 1.000000, Last Sequence Number:2172

Path: 3, Current State: Running, Session ID: 1. Latency: 1.251911, Jitter 0.324650,
˓→Pkt Loss: 0.000000, Jitter Miss: 1.000000, Last Sequence Number:2172

Jitter Appid List:

Latency Appid List:
PacketLoss Appid List:

[root@nsg-2-231-64-98 nuage]# ovs-appctl -t nuage_perfd analyzer/dump 73.106.59.247

UUID DPID PATH_
˓→RUNNING PATH_STOPPED

a31e762c-1bc6-40cb-8219-5f96fdcce666 73.106.59.247
˓→ 4 0

[root@nsg-2-231-64-98 nuage]# ovs-appctl -t nuage_perfd analyzer/dump 73.106.59.247

˓→ details
Path: 0, Current State: Running, Session ID: 1. Latency: 1.039848, Jitter 0.543496,
˓→Pkt Loss: 0.000000, Jitter Miss: 1.000000, Last Sequence Number:2288

Path: 1, Current State: Running, Session ID: 1. Latency: 0.487469, Jitter 0.049811,
˓→Pkt Loss: 0.000000, Jitter Miss: 1.000000, Last Sequence Number:2288

Path: 2, Current State: Running, Session ID: 1. Latency: 0.785121, Jitter 0.177432,
˓→Pkt Loss: 0.000000, Jitter Miss: 1.000000, Last Sequence Number:2288

Path: 3, Current State: Running, Session ID: 1. Latency: 1.251326, Jitter 0.306431,
˓→Pkt Loss: 0.000000, Jitter Miss: 1.000000, Last Sequence Number:2288

Jitter Appid List:

Latency Appid List:
PacketLoss Appid List:

[root@nsg-73-106-59-247 nuage]# ovs-appctl -t nuage_perfd analyzer/dump 4908104c-

˓→1749-4601-8845-daab95eb810f 103.126.194.189 details

Path: 0, Current State: Running, Session ID: 1. Latency: 2.088450, Jitter 0.412167,
˓→Pkt Loss: 0.000000, Jitter Miss: 2.000000, Last Sequence Number:1006

Path: 1, Current State: Running, Session ID: 1. Latency: 1.791934, Jitter 0.402358,
˓→Pkt Loss: 0.000000, Jitter Miss: 2.000000, Last Sequence Number:1006

Path: 2, Current State: Running, Session ID: 1. Latency: 1.639463, Jitter 0.289848,
˓→Pkt Loss: 0.000000, Jitter Miss: 2.000000, Last Sequence Number:1006

Path: 3, Current State: Running, Session ID: 1. Latency: 2.294285, Jitter 0.261720,
˓→Pkt Loss: 0.000000, Jitter Miss: 2.000000, Last Sequence Number:1006

8.1. AAR 484

 VNS User Guide, Release 5.3.2

Jitter Appid List:

Latency Appid List:
Metric: 50.000000 APP_UUID : 4d237281-6e4e-4548-a9d3-d350b3aa9f98,
PacketLoss Appid List:
Metric: 100.000000 APP_UUID : 4d237281-6e4e-4548-a9d3-d350b3aa9f98,

[root@nsg-73-106-59-247 nuage]# ovs-appctl -t nuage_perfd analyzer/dump type=APM

UUID DPID PATH_
˓→RUNNING PATH_STOPPED

4908104c-1749-4601-8845-daab95eb810f 103.126.194.189
˓→ 4 0

4908104c-1749-4601-8845-daab95eb810f 2.231.64.98
˓→ 4 0

[root@nsg-2-231-64-98 nuage]# ovs-appctl -t nuage_perfd analyzer/dump type=NPM

˓→subtype=NAT_IPSEC

UUID DPID PATH_

˓→RUNNING PATH_STOPPED

4eaa6c3c-7de5-42fa-9ba9-7943f4e95313 73.106.59.247
˓→ 4 0

[root@nsg-2-231-64-98 nuage]# ovs-appctl -t nuage_perfd analyzer/dump type=NPM

˓→ subtype=NAT_VXLAN
UUID DPID PATH_
˓→RUNNING PATH_STOPPED

37d34d4d-b950-49d9-8509-51d937e23ef1 73.106.59.247
˓→ 4 0

Neighbor command samples

The ovs-appctl -t nuage_perfd local/dump command displays the local NSG information, such as its
personality, and details regarding network uplinks, such as priority, underlay details and flags.
Uplink Priority values correspond to the following configuration:
1 - Primary1
2 - Primary2
3 - Secondary1
4 - Secondary2
[root@nsg-2-231-64-98 nuage]# ovs-appctl -t nuage_perfd local/dump

Dpid: 2.231.64.98 Personality : NSG_PERSONALITY

Uplink Ip: 10.15.33.254
Priority: 1
Underlay id: 0
Is UBR uplink: false
Is functional: true
Enable NAT Probes: true
Traffic Through UBR Only: false

Uplink Ip: 10.18.33.254

8.1. AAR 485

 VNS User Guide, Release 5.3.2

Priority: 4
Underlay id: 0
Is UBR uplink: false
Is functional: true
Enable NAT Probes: true
Traffic Through UBR Only: false

[root@nsg-2-231-64-98 nuage]# ovs-appctl -t nuage_perfd local/dump json | python -m

˓→ json.tool
{
"LOCAL_NSG_IP": "2.231.64.98",
"Local_Uplink_Info": [
{
"Ip": "10.15.33.254",
"Is_Functional": "true",
"Is_UBR": "false",
"Priority": 1,
"Underlay_Id": 0,
"Uplink": 0
},
{
"Ip": "10.18.33.254",
"Is_Functional": "true",
"Is_UBR": "false",
"Priority": 4,
"Underlay_Id": 0,
"Uplink": 1
}
],
"Personality": "NSG_PERSONALITY",
"Update_Seq_No": 2
}

The ovs-appctl -t nuage_perfd neighbor/dump command displays all the remote NSG information,
such as personality, and details regarding network uplinks, such as priority, underlay details and flags.
Uplink Priority values correspond to the following configuration:
1 - Primary1
2 - Primary2
3 - Secondary1
4 - Secondary2

[root@nsg-2-231-64-98 nuage]# ovs-appctl -t nuage_perfd neighbor/dump

Dpid: 70.127.142.214 Domain:

Uplink Ip: 10.15.1.254,
Priority: 1
Underlay Id: 0
Is UBR Uplink: false
Is Functional: false

Uplink Ip: 10.18.1.254,

Priority: 4
Underlay Id: 0
Is UBR Uplink: false

8.1. AAR 486

 VNS User Guide, Release 5.3.2

Is Functional: false

Dpid: 73.106.59.247 Domain:

Uplink Ip: 10.15.3.254,
Priority: 1
Underlay Id: 0
Is UBR Uplink: false
Is Functional: true

Uplink Ip: 10.18.3.254,

Priority: 4
Underlay Id: 0
Is UBR Uplink: false
Is Functional: true

Dpid: 103.126.194.189 Domain:

Uplink Ip: 10.15.2.254,
Priority: 1
Underlay Id: 0
Is UBR Uplink: false
Is Functional: true

Uplink Ip: 10.18.2.254,

Priority: 4
Underlay Id: 0
Is UBR Uplink: false
Is Functional: true

[root@nsg-2-231-64-98 nuage]# ovs-appctl -t nuage_perfd neighbor/dump json | python -

˓→m json.tool

[
{
"Local_Update_Seq_No": 2,
"Neighbor_IP": "70.127.142.214",
"Neighbor_Uplink_Info": [
{
"Ip": "10.15.1.254",
"Is_Functional": "true",
"Is_UBR": "false",
"Priority": 1,
"Underlay_Id": 0,
"Uplink": 0
},
{
"Ip": "10.18.1.254",
"Is_Functional": "true",
"Is_UBR": "false",
"Priority": 4,
"Underlay_Id": 0,
"Uplink": 1
}
],
"Tx_Path_Info": [
{
"Dst_Port": 50000,
"Is_UBR": "false",
"Path_State": 1,
"Src_Port": 50000

8.1. AAR 487

 VNS User Guide, Release 5.3.2

},
{
"Dst_Port": 50003,
"Is_UBR": "false",
"Path_State": 1,
"Src_Port": 50000
},
{
"Dst_Port": 50000,
"Is_UBR": "false",
"Path_State": 1,
"Src_Port": 50003
},
{
"Dst_Port": 50003,
"Is_UBR": "false",
"Path_State": 1,
"Src_Port": 50003
}
]
},
{
"Local_Update_Seq_No": 2,
"Neighbor_IP": "73.106.59.247",
"Neighbor_Uplink_Info": [
{
"Ip": "10.15.3.254",
"Is_Functional": "true",
"Is_UBR": "false",
"Priority": 1,
"Underlay_Id": 0,
"Uplink": 0
},
{
"Ip": "10.18.3.254",
"Is_Functional": "true",
"Is_UBR": "false",
"Priority": 4,
"Underlay_Id": 0,
"Uplink": 1
}
],
"Tx_Path_Info": [
{
"Dst_Port": 50000,
"Is_UBR": "false",
"Path_State": 1,
"Src_Port": 50000
},
{
"Dst_Port": 50003,
"Is_UBR": "false",
"Path_State": 1,
"Src_Port": 50000
},
{
"Dst_Port": 50000,
"Is_UBR": "false",

8.1. AAR 488

 VNS User Guide, Release 5.3.2

"Path_State": 1,
"Src_Port": 50003
},
{
"Dst_Port": 50003,
"Is_UBR": "false",
"Path_State": 1,
"Src_Port": 50003
}
]
}
]

XPMG command samples

The ovs-appctl -t nuage_perfd xpmg/dump command displays the NPM and APM configuration associ-
ated with this NSG.
Optional arguments to filter the output based on NPM and APM are --npmg and --apmg. The details suffix
provides additional details related to the configuration.
ovs-appctl -t nuage_perfd xpmg/dump --npmg
ovs-appctl -t nuage_perfd xpmg/dump --npmg details
ovs-appctl -t nuage_perfd xpmg/dump --apmg
ovs-appctl -t nuage_perfd xpmg/dump --apmg details

[root@nsg-73-106-59-247 nuage]# ovs-appctl -t nuage_perfd xpmg/dump

APMG_UUID Probe_DSCP Probe_MTU Probe_RATE

7d628590-60c7-4433-bf18-32b65b93ae4d 0 0.000000 0

a31e762c-1bc6-40cb-8219-5f96fdcce666 0 10.000000 500

[root@nsg-73-106-59-247 nuage]# ovs-appctl -t nuage_perfd xpmg/dump --npmg

NPMG_UUID Probe_DSCP Probe_MTU Probe_RATE

a31e762c-1bc6-40cb-8219-5f96fdcce666 0 10.000000 500

[root@nsg-73-106-59-247 nuage]# ovs-appctl -t nuage_perfd xpmg/dump --npmg details

NPMG_UUID: a31e762c-1bc6-40cb-8219-5f96fdcce666
Probe Configuration:
Dscp: 0
Rate: 10.000000
Mtu: 500

[root@nsg-73-106-59-247 nuage]# ovs-appctl -t nuage_perfd xpmg/dump json

[
{
"APMG_UUID": "4908104c-1749-4601-8845-daab95eb810f",
"APPS": [
{
"APMG_UUID": "4908104c-1749-4601-8845-daab95eb810f",
"APM_Trigger": 1,
"APM_UUID": "d9d6227d-da35-4097-bfa5-8bd56309dbe5",
"Is_AAR_Enabled": "true",

8.1. AAR 489

 VNS User Guide, Release 5.3.2

"Jitter": -1,
"Latency": -1,
"Pending_Action": 0,
"Pkt_Loss": 10
},
{
"APMG_UUID": "4908104c-1749-4601-8845-daab95eb810f",
"APM_Trigger": 1,
"APM_UUID": "4d237281-6e4e-4548-a9d3-d350b3aa9f98",
"Is_AAR_Enabled": "true",
"Jitter": -1,
"Latency": 50,
"Pending_Action": 0,
"Pkt_Loss": -1
}
],
"Probe_Config": {
"Dscp": 0,
"Mtu": 576,
"Rate": 0.100000001490116
}
},
{
"NPMG_UUID": "37d34d4d-b950-49d9-8509-51d937e23ef1",
"Probe_Config": {
"Dscp": 46,
"Mtu": 137,
"Rate": 0.100000001490116
}
},
{
"NPMG_UUID": "4eaa6c3c-7de5-42fa-9ba9-7943f4e95313",
"Probe_Config": {
"Dscp": 46,
"Mtu": 137,
"Rate": 0.100000001490116
}
},
{
"APMG_UUID": "7d628590-60c7-4433-bf18-32b65b93ae4d",
"APPS": [
{
"APMG_UUID": "7d628590-60c7-4433-bf18-32b65b93ae4d",
"APM_Trigger": 0,
"APM_UUID": "9cb549e9-fdbc-4142-b0fd-c7c84111dd10",
"Is_AAR_Enabled": "false",
"Jitter": -1,
"Latency": -1,
"Pending_Action": 0,
"Pkt_Loss": -1
}
],
"Probe_Config": {
"Dscp": 0,
"Mtu": 0,
"Rate": 0
}
}

8.1. AAR 490

 VNS User Guide, Release 5.3.2

[root@nsg-73-106-59-247 nuage]# ovs-appctl -t nuage_perfd xpmg/dump --apmg

APMG_UUID Probe_DSCP Probe_MTU Probe_RATE

4908104c-1749-4601-8845-daab95eb810f 0 0.100000 576

7d628590-60c7-4433-bf18-32b65b93ae4d 0 0.000000 0

[root@nsg-73-106-59-247 nuage]# ovs-appctl -t nuage_perfd xpmg/dump --apmg details

APMG_UUID: 4908104c-1749-4601-8845-daab95eb810f
Probe Configuration:
Dscp: 0
Rate: 0.100000
Mtu: 576

Applications:
APM_UUID: 4d237281-6e4e-4548-a9d3-d350b3aa9f98
Is AAR Enabled: true
Latency: 50.000000
Jitter: -1.000000
Pkt Loss: -1.000000
Pending Action: 0
APM Trigger: 1

APMG_UUID: 7d628590-60c7-4433-bf18-32b65b93ae4d
Probe Configuration:
Dscp: 0
Rate: 0.000000
Mtu: 0

Applications:
APM_UUID: 9cb549e9-fdbc-4142-b0fd-c7c84111dd10
Is AAR Enabled: false
Latency: -1.000000
Jitter: -1.000000
Pkt Loss: -1.000000
Pending Action: 0
APM Trigger: 0

[root@nsg-148-189-86-167 ~]# ovs-appctl -t nuage_perfd probe_config/dump

PROBE_UUID: cdc5f147-6208-4402-b6e3-8f1400b5fbf7
Dscp : 0, Rate : 0.000000 packets/s, Mtu : 0 bytes

PROBE_UUID: 17d81fa3-eddb-4112-8693-48a915478b72
Dscp : 0, Rate : 0.100000 packets/s, Mtu : 600 bytes

Log command samples

The vlog/list and vlog/set commands list the module names related to AAR and their logging mode in SYS-
LOG, FILE and CONSOLE.

8.1. AAR 491

 VNS User Guide, Release 5.3.2

ovs-appctl -t nuage_perfd vlog/list | grep aar

ovs-appctl -t nuage_perfd vlog/set <Module_name>:<file|syslog|console|all>:
˓→<DBG|INFO|WARN|ERR>

ovs-appctl -t nuage_perfd vlog/set aar_analyzer:file:INFO

ovs-appctl -t nuage_perfd vlog/set aar_analyzer:all:ERR
ovs-appctl -t nuage_perfd vlog/set aar_analyzer:file:WARN

[root@nsg-73-106-59-247 nuage]# ovs-appctl -t nuage_perfd vlog/list | grep aar

aar_analyzer OFF ERR INFO
aar_anlz_thread OFF ERR INFO
aar_appctl_commands OFF ERR INFO
aar_data_probe_msg OFF ERR INFO
aar_evt_log OFF ERR INFO
aar_generator OFF ERR INFO
aar_hpng_main OFF ERR INFO
aar_hpng_ping OFF ERR INFO
aar_hpng_redis OFF ERR INFO
aar_hpng_test OFF ERR INFO
aar_msg_type OFF ERR INFO
aar_multi_work_queue OFF ERR INFO
aar_nuage_perf_main OFF ERR INFO
aar_nuage_perf_mem OFF ERR INFO
aar_ovsdb OFF ERR INFO
aar_perf_util OFF ERR INFO
aar_probe_validator OFF ERR INFO
aar_publish_stats OFF ERR INFO
aar_redis_flow OFF ERR INFO
aar_redis_handler OFF ERR INFO
aar_rx_api OFF ERR INFO
aar_rx_api_test OFF ERR INFO
aar_rx_thread OFF ERR INFO
aar_ssl_client OFF ERR INFO
aar_test_generator OFF ERR INFO
aar_test_tx_thread OFF ERR INFO
aar_tx_apmg OFF ERR INFO
aar_tx_local OFF ERR INFO
aar_tx_main OFF ERR INFO
aar_tx_neighbor OFF ERR INFO
aar_tx_npmg OFF ERR INFO
aar_tx_path OFF ERR INFO
aar_tx_sock OFF ERR INFO
aar_tx_thread OFF ERR INFO
aar_tx_ubr OFF ERR INFO
aar_tx_xpmg OFF ERR INFO

The event-log command can be used to display or flush the event logs.
ovs-appctl -t nuage_perfd event-log/dump
ovs-appctl -t nuage_perfd event-log/flush

[root@ovs-1 nuage]# ovs-appctl -t nuage_perfd event-log/dump

Event log has 35 entries
Event log start:
======================================================
2017-05-26 21:45:18.321 | AAR_EVT_LOG_MOD_REDIS | REDIS_ADD_LOCAL | 0 | 43.24.132.
˓→192,10.15.1.254:1:0:0,0.0.0.0,0.0.0.0,10.18.1.254:4:0:0,

2017-05-26 21:45:18.321 | AAR_EVT_LOG_MOD_REDIS | REDIS_ADD_NEIGHBOR | 1 | 122.147.7.

˓→202,10.15.3.254:1:0:0,0.0.0.0,0.0.0.0,10.18.3.254:4:0:

˓→0,263277056,1127796765,1883871570,

8.1. AAR 492

 VNS User Guide, Release 5.3.2

2017-05-26 21:45:18.321 | AAR_EVT_LOG_MOD_REDIS | REDIS_ADD_NEIGHBOR | 2 | 116.180.

˓→212.126,10.15.2.254:1:0:0,0.0.0.0,0.0.0.0,0.0.0.0,1127796765,1883871570,

2017-05-26 21:45:18.321 | AAR_EVT_LOG_MOD_REDIS | REDIS_ADD_NEIGHBOR | 3 | 71.86.165.

˓→6,10.15.33.254:1:0:0,0.0.0.0,0.0.0.0,0.0.0.0,263277056,1127796765,

2017-05-26 21:45:18.325 | AAR_EVT_LOG_MOD_OVSDB | OVSDB_ADD_NPMG | 0 | NPMG 37a65c05-

˓→2acf-4f7a-91af-600b62b3fe2b

2017-05-26 21:45:18.325 | AAR_EVT_LOG_MOD_OVSDB | OVSDB_ADD_NPMG | 1 | NPMG 376f9107-

˓→5fcd-4d9f-9901-1fa8657a782c

2017-05-26 21:45:18.325 | AAR_EVT_LOG_MOD_OVSDB | OVSDB_ADD_XPMG_DPID | 2 | xpmg-

˓→neighbor xpmg uuid: 37a65c05-2acf-4f7a-91af-600b62b3fe2b, app uuid: 00000000-0000-

˓→0000-0000-000000000000, dpid: 122.147.7.202

2017-05-26 21:45:18.325 | AAR_EVT_LOG_MOD_OVSDB | OVSDB_ADD_XPMG_DPID | 3 | xpmg-

˓→neighbor xpmg uuid: 376f9107-5fcd-4d9f-9901-1fa8657a782c, app uuid: 00000000-0000-

˓→0000-0000-000000000000, dpid: 122.147.7.202

2017-05-26 21:45:18.325 | AAR_EVT_LOG_MOD_OVSDB | OVSDB_ADD_XPMG_DPID | 4 | xpmg-

˓→neighbor xpmg uuid: 37a65c05-2acf-4f7a-91af-600b62b3fe2b, app uuid: 00000000-0000-

˓→0000-0000-000000000000, dpid: 71.86.165.6

2017-05-26 21:45:18.325 | AAR_EVT_LOG_MOD_OVSDB | OVSDB_ADD_XPMG_DPID | 5 | xpmg-

˓→neighbor xpmg uuid: 37a65c05-2acf-4f7a-91af-600b62b3fe2b, app uuid: 00000000-0000-

˓→0000-0000-000000000000, dpid: 116.180.212.126

2017-05-26 21:45:20.328 | AAR_EVT_LOG_MOD_ROUTE | ROUTE_ADD_LOCAL | 0 | 43.24.132.192

2017-05-26 21:45:20.329 | AAR_EVT_LOG_MOD_ROUTE | ROUTE_ADD_NEIGHBOR | 1 | 116.180.
˓→212.126

2017-05-26 21:45:20.329 | AAR_EVT_LOG_MOD_ROUTE | ROUTE_ADD_NEIGHBOR | 2 | 71.86.165.6

2017-05-26 21:45:20.329 | AAR_EVT_LOG_MOD_ROUTE | ROUTE_ADD_NEIGHBOR | 3 | 122.147.7.
˓→202

2017-05-26 21:45:20.329 | AAR_EVT_LOG_MOD_CONFIG | CONFIG_ADD_NPMG | 0 | 37a65c05-

˓→2acf-4f7a-91af-600b62b3fe2b

2017-05-26 21:45:20.329 | AAR_EVT_LOG_MOD_CONFIG | CONFIG_ADD_NPMG | 1 | 376f9107-

˓→5fcd-4d9f-9901-1fa8657a782c

2017-05-26 21:45:20.329 | AAR_EVT_LOG_MOD_ROUTE | CONFIG_ADD_XPMG_DPID | 4 | 37a65c05-

˓→2acf-4f7a-91af-600b62b3fe2b 122.147.7.202

2017-05-26 21:45:20.329 | AAR_EVT_LOG_MOD_GNR | GNR_START | 0 | XPMG UUID : 37a65c05-

˓→2acf-4f7a-91af-600b62b3fe2b, Dpid: 122.147.7.202, state = 0

2017-05-26 21:45:20.330 | AAR_EVT_LOG_MOD_ROUTE | CONFIG_ADD_XPMG_DPID | 5 | 376f9107-

˓→5fcd-4d9f-9901-1fa8657a782c 122.147.7.202

2017-05-26 21:45:20.330 | AAR_EVT_LOG_MOD_GNR | GNR_START | 1 | XPMG UUID : 376f9107-

˓→5fcd-4d9f-9901-1fa8657a782c, Dpid: 122.147.7.202, state = 0

2017-05-26 21:45:20.330 | AAR_EVT_LOG_MOD_ROUTE | CONFIG_ADD_XPMG_DPID | 6 | 37a65c05-

˓→2acf-4f7a-91af-600b62b3fe2b 71.86.165.6

2017-05-26 21:45:20.330 | AAR_EVT_LOG_MOD_GNR | GNR_START | 2 | XPMG UUID : 37a65c05-

˓→2acf-4f7a-91af-600b62b3fe2b, Dpid: 71.86.165.6, state = 0

2017-05-26 21:45:20.330 | AAR_EVT_LOG_MOD_ROUTE | CONFIG_ADD_XPMG_DPID | 7 | 37a65c05-

˓→2acf-4f7a-91af-600b62b3fe2b 116.180.212.126

2017-05-26 21:45:20.330 | AAR_EVT_LOG_MOD_GNR | GNR_START | 3 | XPMG UUID : 37a65c05-

˓→2acf-4f7a-91af-600b62b3fe2b, Dpid: 116.180.212.126, state = 0

2017-05-26 21:48:43.420 | AAR_EVT_LOG_MOD_OVSDB | OVSDB_ADD_NPMG | 6 | NPMG a21e129c-

˓→0c6d-4d1d-99a3-87801bacc13f

2017-05-26 21:48:43.420 | AAR_EVT_LOG_MOD_CONFIG | CONFIG_ADD_NPMG | 2 | a21e129c-

˓→0c6d-4d1d-99a3-87801bacc13f

2017-05-26 21:48:47.246 | AAR_EVT_LOG_MOD_OVSDB | OVSDB_ADD_XPMG_DPID | 7 | xpmg-

˓→neighbor xpmg uuid: a21e129c-0c6d-4d1d-99a3-87801bacc13f, app uuid: 00000000-0000-

˓→0000-0000-000000000000, dpid: 116.180.212.126

2017-05-26 21:48:47.246 | AAR_EVT_LOG_MOD_ROUTE | CONFIG_ADD_XPMG_DPID | 8 | a21e129c-

˓→0c6d-4d1d-99a3-87801bacc13f 116.180.212.126

2017-05-26 21:48:47.246 | AAR_EVT_LOG_MOD_GNR | GNR_START | 4 | XPMG UUID : a21e129c-

˓→0c6d-4d1d-99a3-87801bacc13f, Dpid: 116.180.212.126, state = 0

2017-05-26 21:48:47.247 | AAR_EVT_LOG_MOD_OVSDB | OVSDB_ADD_XPMG_DPID | 8 | xpmg-

˓→neighbor xpmg uuid: a21e129c-0c6d-4d1d-99a3-87801bacc13f, app uuid: 00000000-0000-

˓→0000-0000-000000000000, dpid: 71.86.165.6

8.1. AAR 493

 VNS User Guide, Release 5.3.2

2017-05-26 21:48:47.247 | AAR_EVT_LOG_MOD_ROUTE | CONFIG_ADD_XPMG_DPID | 9 | a21e129c-

˓→0c6d-4d1d-99a3-87801bacc13f 71.86.165.6

2017-05-26 21:48:47.247 | AAR_EVT_LOG_MOD_GNR | GNR_START | 5 | XPMG UUID : a21e129c-

˓→0c6d-4d1d-99a3-87801bacc13f, Dpid: 71.86.165.6, state = 0

2017-05-26 21:48:47.247 | AAR_EVT_LOG_MOD_OVSDB | OVSDB_ADD_XPMG_DPID | 9 | xpmg-

˓→neighbor xpmg uuid: a21e129c-0c6d-4d1d-99a3-87801bacc13f, app uuid: 00000000-0000-

˓→0000-0000-000000000000, dpid: 122.147.7.202

2017-05-26 21:48:47.247 | AAR_EVT_LOG_MOD_ROUTE | CONFIG_ADD_XPMG_DPID | 10 |

˓→a21e129c-0c6d-4d1d-99a3-87801bacc13f 122.147.7.202

2017-05-26 21:48:47.247 | AAR_EVT_LOG_MOD_GNR | GNR_START | 6 | XPMG UUID : a21e129c-

˓→0c6d-4d1d-99a3-87801bacc13f, Dpid: 122.147.7.202, state = 0

======================================================
[root@ovs-1 nuage]# ovs-appctl -t nuage_perfd event-log/flush
Copy the event file while calling flush.
[root@ovs-1 nuage]# ovs-appctl -t nuage_perfd event-log/flush >> /home/nuage/event_
˓→log_file

[root@ovs-1 nuage]# ovs-appctl -t nuage_perfd event-log/dump

Event log has 0 entries
Event log start:
======================================================
======================================================
Event log ends.

Application command samples

The ovs-appctl -t nuage_perfd app/dump command displays a list of all APMGs and associated appli-
cations. The IS_AAR_ENABLED field shows whether AAR is enabled for the APMG. The Latency, Jitter, and
Packet_Loss fields show the performance statistics for the APMG.
If you specify an APMG UUID as part of the command, the output lists only the specified APMG.

[root@nsg-77-34-172-211 ~]# ovs-appctl -t nuage_perfd app/dump

Applications Dump

APMG UUID: 294266c7-6f95-41ec-9f30-322738060df4

APM UUID: 9eaab2f3-7639-465f-98a3-8dba9ed6711c
Is AAR Enabled: true
Latency: 22.000000
Jitter: 22.000000
Pkt Loss: 2.000000
Pending Action: 0
APM Trigger: 1

APMG UUID: 34af5a6c-5c25-4e05-a0e6-c5fd863661b1

APM UUID: fe00cb1e-1569-4143-ae91-4329641409b0
Is AAR Enabled: false
Latency: -1.000000
Jitter: -1.000000
Pkt Loss: -1.000000
Pending Action: 0
APM Trigger: 0

[root@nsg-77-34-172-211 ~]# ovs-appctl -t nuage_perfd app/dump 294266c7-6f95-41ec-

˓→9f30-322738060df4

Applications Dump

8.1. AAR 494

 VNS User Guide, Release 5.3.2

APMG UUID: 294266c7-6f95-41ec-9f30-322738060df4

APM UUID: 9eaab2f3-7639-465f-98a3-8dba9ed6711c
Is AAR Enabled: true
Latency: 22.000000
Jitter: 22.000000
Pkt Loss: 2.000000
Pending Action: 0
APM Trigger: 1

[root@nsg-180-98-7-143 ~]# ovs-appctl -t nuage_perfd app/dump json | python -m json.

˓→tool

{
"Application Dump": [
{
"APMG_UUID": "b6572acd-372e-4235-98cc-20997bd04c51",
"APM_Trigger": 1,
"APM_UUID": "573c8425-5251-422a-a9dd-aa28c69e0acb",
"Is_AAR_Enabled": "true",
"Jitter": 22,
"Latency": 22,
"Pending_Action": 0,
"Pkt_Loss": 2
},
{
"APMG_UUID": "b6572acd-372e-4235-98cc-20997bd04c51",
"APM_Trigger": 1,
"APM_UUID": "411fce2b-d88b-4c04-8590-90764d04bf3d",
"Is_AAR_Enabled": "true",
"Jitter": 20,
"Latency": 20,
"Pending_Action": 0,
"Pkt_Loss": 2
},
{
"APMG_UUID": "e0de46a6-1b30-4a7f-ba26-ae5c00e84b7e",
"APM_Trigger": 0,
"APM_UUID": "2eec0c68-b4ef-433d-ac80-516782d1f393",
"Is_AAR_Enabled": "false",
"Jitter": -1,
"Latency": -1,
"Pending_Action": 0,
"Pkt_Loss": -1
}
]
}

SLA violation command samples

The ovs-appctl -t nuage_perfd slaviolations/dump command displays the SLA Violation cache
with details such as sorted list of Paths and violation type on paths.
Path values represent uplink pair combinations:
Path value 0 : local newtork uplink 1 to remote newtork uplink 1
Path value 1 : local newtork uplink 2 to remote newtork uplink 1
Path value 2 : local newtork uplink 1 to remote newtork uplink 2

8.1. AAR 495

 VNS User Guide, Release 5.3.2

Path value 3 : local newtork uplink 2 to remote newtork uplink 2

Violation Type values to violation type:
Violation type 1 : Latency is violated
Violation type 2 : Jitter is violated
Violation type 3 : Latency and Jitter are violated
Violation type 4 : Packet Loss is violated
Violation type 5 : Packet Loss and Latecy are violated
Violation type 6 : Packet Loss and Jitter are violated
Violation type 7 : Latency, Packet Loss and Jitter are violated
[root@nsg-77-34-172-211 ~]# ovs-appctl -t nuage_perfd slaviolations/dump

Generator XPMG ID: 294266c7-6f95-41ec-9f30-322738060df4

DPID: 222.130.32.92
APP UUID: 9eaab2f3-7639-465f-98a3-8dba9ed6711c
Sorted_Path [0]: 3
Sorted_Path [1]: 0
Sorted_Path [2]: 2
Sorted_Path [3]: 1
Path 0: Violation_type 0
Path 1: Violation_type 1
Path 2: Violation_type 0
Path 3: Violation_type 0

[root@nsg-180-98-7-143 ~]# ovs-appctl -t nuage_perfd slaviolations/dump b6572acd-372e-

˓→4235-98cc-20997bd04c51 68.125.142.225

Generator XPMG ID: 294266c7-6f95-41ec-9f30-322738060df4

DPID: 222.130.32.92
APP UUID: 9eaab2f3-7639-465f-98a3-8dba9ed6711c
Sorted_Path [0]: 3
Sorted_Path [1]: 0
Sorted_Path [2]: 2
Sorted_Path [3]: 1
Path 0: Violation_type 0
Path 1: Violation_type 1
Path 2: Violation_type 0
Path 3: Violation_type 0

[root@nsg-77-34-172-211 ~]# ovs-appctl -t nuage_perfd slaviolations/dump json |

˓→python -m json.tool

{
"Sla Violations": [
{
"App_Id": "9eaab2f3-7639-465f-98a3-8dba9ed6711c",
"Path Violation Type": [
0,
1,
0,
0
],
"Sorted Paths": [
3,

8.1. AAR 496

 VNS User Guide, Release 5.3.2

0,
2,
1
]
}
]
}

NSG-UBR command samples

The ovs-appctl -t nuage_perfd ubr/dump command displays the sorted list of paths through UBRs based
on Delay and Packet Loss On a NSG personality.
[root@nsg-154-179-79-109 nuage]# ovs-appctl -t nuage_perfd ubr/dump
NSG_UBR_DPID PATH DELAY NSG_UBR_PRIORITY
˓→ VIOLATED DELAY_VIOLATED PKTLOSS_VIOLATED PUBLISHED
˓→ OVS_PRIORITY
171.116.116.133 1 1.850210 100
˓→ FALSE FALSE FALSE FALSE
˓→ 3999998000
111.111.175.173 1 1.970886 100
˓→ FALSE FALSE FALSE FALSE
˓→ 3999997500
81.78.97.229 0 2.979409 100
˓→ FALSE FALSE FALSE TRUE
˓→ 3999994000
133.76.62.161 0 1.481657 100
˓→ FALSE FALSE FALSE FALSE
˓→ 3999999000
133.76.62.161 3 1.039034 100
˓→ FALSE FALSE FALSE FALSE
˓→ 4000000000
171.116.116.133 2 2.311163 100
˓→ FALSE FALSE FALSE TRUE
˓→ 3999997000
111.111.175.173 2 2.416106 100
˓→ FALSE FALSE FALSE TRUE
˓→ 3999996000
81.78.97.229 3 2.597735 100
˓→ FALSE FALSE FALSE FALSE
˓→ 3999995000

[root@nsg-154-179-79-109 nuage]# ovs-appctl -t nuage_perfd ubr/dump json | python -m

˓→json.tool

[
{
"NSG_Info":[
{
"Path":0,
"Is_Delay_Violated":"FALSE",
"Is_PktLoss_Violated":"FALSE",
"Nsg_List_Priority":4000000000,
"IP":"142.185.147.184",
"Is_Violated":"FALSE",
"Ovs_List_Priority":4000000000,
"Is_Published":"TRUE",
"Delay":1.94742834568024
},

8.1. AAR 497

 VNS User Guide, Release 5.3.2

{
"Path":3,
"Is_Delay_Violated":"FALSE",
"Is_PktLoss_Violated":"FALSE",
"Nsg_List_Priority":3999999000,
"IP":"142.185.147.184",
"Is_Violated":"FALSE",
"Ovs_List_Priority":3999999000,
"Is_Published":"TRUE",
"Delay":2.365807056427
}
]
},
{
"NSG_Info":[
{
"Path":0,
"Is_Delay_Violated":"FALSE",
"Is_PktLoss_Violated":"FALSE",
"Nsg_List_Priority":4000000000,
"IP":"44.31.79.75",
"Is_Violated":"FALSE",
"Ovs_List_Priority":4000000000,
"Is_Published":"TRUE",
"Delay":1.9617612361908
},
{
"Path":3,
"Is_Delay_Violated":"FALSE",
"Is_PktLoss_Violated":"FALSE",
"Nsg_List_Priority":3999999000,
"IP":"44.31.79.75",
"Is_Violated":"FALSE",
"Ovs_List_Priority":3999999000,
"Is_Published":"TRUE",
"Delay":2.44601345062256
}
]
},
{
"NSG_Info":[
{
"Path":3,
"Is_Delay_Violated":"FALSE",
"Is_PktLoss_Violated":"FALSE",
"Nsg_List_Priority":4000000000,
"IP":"154.179.79.109",
"Is_Violated":"FALSE",
"Ovs_List_Priority":4000000000,
"Is_Published":"TRUE",
"Delay":1.04982900619507
},
{
"Path":0,
"Is_Delay_Violated":"FALSE",
"Is_PktLoss_Violated":"FALSE",
"Nsg_List_Priority":3999999000,
"IP":"154.179.79.109",

8.1. AAR 498

 VNS User Guide, Release 5.3.2

"Is_Violated":"FALSE",
"Ovs_List_Priority":3999999000,
"Is_Published":"TRUE",
"Delay":1.56927061080933
}
]
},
{
"NSG_Info":[
{
"Path":3,
"Is_Delay_Violated":"FALSE",
"Is_PktLoss_Violated":"FALSE",
"Nsg_List_Priority":4000000000,
"IP":"163.212.74.4",
"Is_Violated":"FALSE",
"Ovs_List_Priority":4000000000,
"Is_Published":"TRUE",
"Delay":2.58935570716858
},
{
"Path":0,
"Is_Delay_Violated":"FALSE",
"Is_PktLoss_Violated":"FALSE",
"Nsg_List_Priority":3999999000,
"IP":"163.212.74.4",
"Is_Violated":"FALSE",
"Ovs_List_Priority":3999999000,
"Is_Published":"TRUE",
"Delay":3.02561092376709
}
]
}
]

The ovs-appctl -t nuage_perfd ubr_ovs_published/dump command displays the sorted list of paths
through UBRs based on Delay and Packet Loss On a NSG personality that has been published to OVS.
[root@nsg-154-179-79-109 nuage]# ovs-appctl -t nuage_perfd ubr_ovs_published/dump
NSG_UBR_DPID PATH DELAY NSG_UBR_PRIORITY
˓→ VIOLATED DELAY_VIOLATED PKTLOSS_VIOLATED PUBLISHED
˓→ OVS_PRIORITY
171.116.116.133 1 1.850210 100
˓→ FALSE FALSE FALSE FALSE
˓→ 3999998000
111.111.175.173 1 1.970886 100
˓→ FALSE FALSE FALSE FALSE
˓→ 3999997500
81.78.97.229 0 2.979409 100
˓→ FALSE FALSE FALSE TRUE
˓→ 3999994000
133.76.62.161 0 1.481657 100
˓→ FALSE FALSE FALSE FALSE
˓→ 3999999000
133.76.62.161 3 1.039034 100
˓→ FALSE FALSE FALSE FALSE
˓→ 4000000000
171.116.116.133 2 2.311163 100
˓→ FALSE FALSE FALSE TRUE
˓→ 3999997000
8.1. AAR 499
 VNS User Guide, Release 5.3.2

111.111.175.173 2 2.416106 100

˓→ FALSE FALSE FALSE TRUE
˓→ 3999996000
81.78.97.229 3 2.597735 100
˓→ FALSE FALSE FALSE FALSE
˓→ 3999995000

The ovs-appctl -t nuage_perfd nsg/dump command displays the sorted list of paths to NSGs based on
Delay and Packet Loss On a UBR personality.

[root@nsg-133-76-62-161 nuage]# ovs-appctl -t nuage_perfd nsg/dump

NSG_DPID PATH DELAY VIOLATED
˓→ DELAY_VIOLATED PKTLOSS_VIOLATED PUBLISHED OVS_
˓→PRIORITY

142.185.147.184 0 1.947428 FALSE

˓→ FALSE FALSE TRUE
˓→4000000000

142.185.147.184 3 2.365807 FALSE

˓→ FALSE FALSE TRUE
˓→3999999000

44.31.79.75 0 1.961761 FALSE

˓→ FALSE FALSE TRUE
˓→4000000000

44.31.79.75 3 2.446013 FALSE

˓→ FALSE FALSE TRUE
˓→3999999000

154.179.79.109 3 1.049829 FALSE

˓→ FALSE FALSE TRUE
˓→4000000000

154.179.79.109 0 1.569271 FALSE

˓→ FALSE FALSE TRUE
˓→3999999000

163.212.74.4 3 2.589356 FALSE

˓→ FALSE FALSE TRUE
˓→4000000000

163.212.74.4 0 3.025611 FALSE

˓→ FALSE FALSE TRUE
˓→3999999000

[root@nsg-133-76-62-161 nuage]# ovs-appctl -t nuage_perfd nsg/dump json | python -m

˓→json.tool

[
{
"NSG_Info":[
{
"Path":0,
"Is_Delay_Violated":"FALSE",
"Is_PktLoss_Violated":"FALSE",
"Nsg_List_Priority":4000000000,
"IP":"142.185.147.184",
"Is_Violated":"FALSE",
"Ovs_List_Priority":4000000000,
"Is_Published":"TRUE",
"Delay":1.94742834568024
},
{
"Path":3,

8.1. AAR 500

 VNS User Guide, Release 5.3.2

"Is_Delay_Violated":"FALSE",
"Is_PktLoss_Violated":"FALSE",
"Nsg_List_Priority":3999999000,
"IP":"142.185.147.184",
"Is_Violated":"FALSE",
"Ovs_List_Priority":3999999000,
"Is_Published":"TRUE",
"Delay":2.365807056427
}
]
},
{
"NSG_Info":[
{
"Path":0,
"Is_Delay_Violated":"FALSE",
"Is_PktLoss_Violated":"FALSE",
"Nsg_List_Priority":4000000000,
"IP":"44.31.79.75",
"Is_Violated":"FALSE",
"Ovs_List_Priority":4000000000,
"Is_Published":"TRUE",
"Delay":1.9617612361908
},
{
"Path":3,
"Is_Delay_Violated":"FALSE",
"Is_PktLoss_Violated":"FALSE",
"Nsg_List_Priority":3999999000,
"IP":"44.31.79.75",
"Is_Violated":"FALSE",
"Ovs_List_Priority":3999999000,
"Is_Published":"TRUE",
"Delay":2.44601345062256
}
]
},
{
"NSG_Info":[
{
"Path":3,
"Is_Delay_Violated":"FALSE",
"Is_PktLoss_Violated":"FALSE",
"Nsg_List_Priority":4000000000,
"IP":"154.179.79.109",
"Is_Violated":"FALSE",
"Ovs_List_Priority":4000000000,
"Is_Published":"TRUE",
"Delay":1.04982900619507
},
{
"Path":0,
"Is_Delay_Violated":"FALSE",
"Is_PktLoss_Violated":"FALSE",
"Nsg_List_Priority":3999999000,
"IP":"154.179.79.109",
"Is_Violated":"FALSE",
"Ovs_List_Priority":3999999000,

8.1. AAR 501

 VNS User Guide, Release 5.3.2

"Is_Published":"TRUE",
"Delay":1.56927061080933
}
]
},
{
"NSG_Info":[
{
"Path":3,
"Is_Delay_Violated":"FALSE",
"Is_PktLoss_Violated":"FALSE",
"Nsg_List_Priority":4000000000,
"IP":"163.212.74.4",
"Is_Violated":"FALSE",
"Ovs_List_Priority":4000000000,
"Is_Published":"TRUE",
"Delay":2.58935570716858
},
{
"Path":0,
"Is_Delay_Violated":"FALSE",
"Is_PktLoss_Violated":"FALSE",
"Nsg_List_Priority":3999999000,
"IP":"163.212.74.4",
"Is_Violated":"FALSE",
"Ovs_List_Priority":3999999000,
"Is_Published":"TRUE",
"Delay":3.02561092376709
}
]
}
]

The ovs-appctl -t nuage_perfd nsg_ovs_published/dump command displays the sorted list of paths
to NSGs based on Delay and Packet Loss published to OVS on a UBR personality.
[root@nsg-133-76-62-161 nuage]# ovs-appctl -t nuage_perfd nsg_ovs_published/dump
NSG_DPID PATH DELAY VIOLATED
˓→ DELAY_VIOLATED PKTLOSS_VIOLATED PUBLISHED OVS_
˓→PRIORITY

142.185.147.184 0 1.947428 FALSE

˓→ FALSE FALSE TRUE
˓→4000000000

142.185.147.184 3 2.365807 FALSE

˓→ FALSE FALSE TRUE
˓→3999999000

44.31.79.75 0 1.961761 FALSE

˓→ FALSE FALSE TRUE
˓→4000000000

44.31.79.75 3 2.446013 FALSE

˓→ FALSE FALSE TRUE
˓→3999999000

154.179.79.109 3 1.049829 FALSE

˓→ FALSE FALSE TRUE
˓→4000000000

154.179.79.109 0 1.569271 FALSE

˓→ FALSE FALSE TRUE
˓→3999999000

8.1. AAR 502

 VNS User Guide, Release 5.3.2

163.212.74.4 3 2.589356 FALSE

˓→ FALSE FALSE TRUE
˓→4000000000

163.212.74.4 0 3.025611 FALSE

˓→ FALSE FALSE TRUE
˓→ 3999999000

8.1.10 DPI Debug CLI

The following commands provide DPI related information. Optional parameters are in brackets, for example [domain-
id].
Dump info of flattened list

ovs-appctl -t nuage-dpi flat-list/show [domain-id]

domain_id app_id app_grp_id

˓→ priority guid src_ip
˓→ dst_ip src_port dst_port l4_proto ether_type dscp cn_
˓→name pre_class post_class is_discovery
674815103 7ba8e14b-13d3-4968-8470-bb6f5cb0e43b d13e3a3b-672c-46f9-ac1f-
˓→e4169161a96f -65536 * *
˓→ * * * -1 -1
˓→-1 * ANY yes
958113167 7ba8e14b-13d3-4968-8470-bb6f5cb0e43b d13e3a3b-672c-46f9-ac1f-
˓→e4169161a96f -65536 * *
˓→ * * * -1 -1
˓→-1 * ANY yes

List of vPorts with DPI ID stored VRF ID

ovs-appctl -t nuage-dpi dpiid/show [dpi_id]

dpi_id vrf_id vport_name ofport is_dirty

3815 674815103 port3.1 8 0
3375 197232700 port3.3 10 0

List of Connection trackers

ovs-appctl -t nuage-dpi conn-tracker/show [connection_id]

conn_id l7_class is_pub ofport vrf_id src_ip

˓→ dst_ip src_port dst_port l4_proto
˓→ether_type dscp direction_id
14838 HTTP Yes 8 1090865872 50.5.0.101
˓→ 50.1.0.101 17337 20480 6
˓→2048 0 0
15368 HTTP Yes 8 1090865872 50.5.0.101
˓→ 50.1.0.101 21947 20480 6
˓→2048 0 0
25500 HTTP Yes 8 1090865872 50.5.0.101
˓→ 50.1.0.101 59874 20480 6
˓→2048 0 0

Dump info from App hash maps

8.1. AAR 503

 VNS User Guide, Release 5.3.2

ovs-appctl -t nuage-dpi app/show [app_uuid]

app_uuid l7_guid
˓→ src_ip dst_ip
˓→ src_port dst_port
˓→ l4_proto ethertype dscp
˓→ cn_name
4b593fee-55fa-41fd-bb1f-94789db5b197 HTTP
˓→ 255.255.255.3-255.255.255.3 255.255.255.255-255.
˓→255.255.255 6533-6533,1220-1230 1-
˓→1,65520-65520 6 0x86DD 0x00
˓→ *
49abeb3a-4be9-4b20-acee-646574369bfc *
˓→ * *
˓→ 5566-5566 4449-4449
˓→ 6 0x0800 *
˓→ *
bd27baca-4c6a-4a28-b380-62f4d9747f10 *
˓→ * *
˓→ * *
˓→ * * *
˓→ *

Dump info from Appgroup hash maps

ovs-appctl -t nuage-dpi appgroup/show [appgroup_uuid]

appgroup_uuid app_uuid
˓→ priority is_processed is_discovery
17d9a08f-9d97-4b11-90d4-3f5e8c9d9647 300705c6-85e5-4386-83ed-5f3ffd694865
˓→ 20 yes yes
ac6e2920-872d-4366-85ed-4d14baa664ad 300705c6-85e5-4386-83ed-5f3ffd694865
˓→ 10 yes yes
8eca7ab0-8874-499f-afc0-e13c9aaf7123 8cb48e3e-4b81-4e8d-81e1-c7391e95d06e
˓→ 0 yes yes

8.1. AAR 504

 VNS User Guide, Release 5.3.2

8.2 AAR Visualization

• Overview (page 507)

• Terminology (page 508)
• Workflows (page 508)
– Access AAR Visualization Reports (page 508)
– Monitor and analyze application usage at the enterprise level (page 509)
– Monitor and analyze application usage at the domain level (page 509)
• Scrolling Graphs (page 510)
• Enterprise Dashboard (page 511)
– SLA Status - Flow effectiveness score (page 511)
– Top 5 Discovered Applications (page 511)
– Top 5 Upload Users (page 511)
– Top 5 Download Users (page 511)
– Newly Discovered Default Application (page 512)
• Enterprise Detail Dashboard (page 512)
– Enterprise Applications (page 512)
– Data Usage on Apps per NSG (page 512)
– Data Usage per NSG for Selected Application (page 512)
– Cumulative Data Usage on NSG for Selected Application (page 513)
– NSG Application Traffic Details for Source IP (page 513)
– Enterprise - Top 20 Applications (page 513)
– Application specific Date Histogram (page 513)
• Enterprise Map View (page 514)
– Map View Prerequisites (page 514)

* Google Maps API Key (page 514)

* NSG Location (page 515)
– Map View Dashboard (page 515)

* Bootstrapping Status (page 516)

* NSG Search Bar (page 517)
* CSV Export (page 517)
• NSG Dashboard (page 518)
– NSG Application Traffic Detail (page 518)
– NSG Application Traffic Details for Application (page 518)
– NSG Application Traffic Details for Source IP (page 518)

8.2. AAR Visualization 505

 VNS User Guide, Release 5.3.2

– Per Port Traffic Visualization (page 519)

– Top 10 Applications (page 519)
– Top 5 Discovered Application - Usage over Time (page 519)
– Top 5 Talkers (Upload) (page 519)
– Top 5 Talkers (Download) (page 519)
– Alarms (page 519)
• NSG Detail Dashboard (page 520)
– From NSG Application Report (page 520)
– To NSG Application Report (page 520)
– From NSG SLA Report (page 520)
– To NSG SLA Report (page 520)
• NAT-T Events Dashboard (page 521)
• IKE Dashboard (page 521)
• Alarms Dashboard (page 521)
• Domain - Applications Dashboard (page 521)
– Top 20 Talkers in Domain (page 521)
– Top 5 Applications in Domain (page 521)
– Top 5 Application Groups (page 522)
• Domain - Network Performance Dashboard (page 522)
– From NSG (page 522)
– To NSG (page 522)
– Probe Detail (NSG as Source) (page 522)
– Probe Detail (NSG as Destination) (page 523)
• Domain - Application Performance Management Dashboard (page 523)
– From NSG (page 524)
– To NSG (page 524)
– SLA HeatMap (page 524)
– Traffic Uplink Pairs for Application (page 524)
– Traffic for Application (page 525)
– SLA Packet Loss for Application (page 525)
– SLA Delay for Application (page 525)
– SLA Jitter for Application (page 525)
• Domain - Details Dashboard (page 525)
– Domain Applications (page 525)
– Data Usage on Apps per NSG (page 526)

8.2. AAR Visualization 506

 VNS User Guide, Release 5.3.2

– Data Usage per NSG for Selected Application (page 526)

– Cumulative Data Usage on NSG for Selected Application (page 526)
– NSG Application Traffic Details for Source IP (page 526)
• Query Examples (page 526)
– Enterprise Dashboard (page 526)

* SLA Status - Flow effectiveness score (page 526)

* Top 5 Applications (page 527)
* Top 5 Upload Users (page 528)
* Top 5 Download Users (page 530)
* Newly Discovered Default Application (page 531)
– Enterprise Detail Dashboard (page 532)

* Enterprise - Top 20 Applications (page 532)

* Application specific Date Histogram (page 534)
– NSG Dashboard (page 535)

* Top 10 Applications (page 535)

* Top 5 Discovered Application - Usage over Time (page 537)
* Top 5 Talkers (Upload) (page 538)
* Top 5 Talkers (download) (page 540)
– NSG Detail Dashboard (page 541)

* From NSG Application Report (page 541)

* To NSG Application Report (page 545)
* From NSG SLA Report (page 550)
* To NSG SLA Report (page 551)
– Domain - Applications Dashboard (page 553)

* Top 20 Talkers in Domain (page 553)

* Top 5 Applications (page 556)
* Top 5 Application Groups (page 558)
– Domain - Network Performance Dashboard (page 559)

* Probe Detail (NSG as Source) (page 559)

* Probe Detail (NSG as Destination) (page 560)

8.2.1 Overview

Visualization is centralized and consumed as a separate application from the VSD UI. The statistics are retrieved from
Elastic Search as part of VSP Collector and Statistics warehousing solution based on JSON queries. Visualizations
may use a single or multiple queries to present the data. Such queries may come from ES or VSD or both.

8.2. AAR Visualization 507

 VNS User Guide, Release 5.3.2

Attention: Refer to Procedure for Installing Statistics for AAR in the Nuage VNS Install Guide for installation
and configuration procedures for enabling AAR visualization.

This chapter describes the different AAR visualization reports along with the associated query examples.

8.2.2 Terminology

• Discovered application: These are applications reported by the NSG that do not have an application defined in
VSD. This is typical of a discover use-case where a user is trying to understand which applications are traversing
the network. Such discovered applications are reported under the default APM Group and default Application.
In this mode, user can identify specific applications by looking at the matched signature field that is reported
in the respective visualizations charts. Layer-7 Classification columns in all detailed visualization reports, will
also display discovered applications with Common Name (from TLS certificates) when available.
• Applications: These are applications reported by the NSG that do have an application defined in VSD. Such
applications are generally known ahead of time, and as such have matching rules, SLAs and probes defined
in VSD. In this mode, user can identify specific applications by looking at the Application Name field that is
reported in the respective visualizations charts.
• Talkers: Includes both discovered applications and applications that have been pre-configured in VSD.
• Users: Include IP addresses that are sources or destination of traffic.

8.2.3 Workflows

This section includes workflows for monitoring and analyzing application usage in a VNS network.

Access AAR Visualization Reports

AAR visualization reports are available at the Enterprise level (assuming that Stats have been enabled on the VSD),
and accessed as follows:
1. In the Enterprise view, navigate to Dashboard –> AAR Statistics tab. This opens a new browser window for
AAR visualization.
2. Determine the ES Cluster Proxy Address from the Statistics section of the Platform System Configuration
(Settings -> System Configuration ).
3. Paste the address:port number (6200) in another browser window and add the security exception for certificates.
4. Navigate back to the AAR visualization window, and refresh it to view the Enterprise Dashboard. Use the menu
option on the upper left corner of the screen (shown in Figure below) for navigating to the various reports. The
dropdown will display all the available report options.

8.2. AAR Visualization 508

 VNS User Guide, Release 5.3.2

Fig. 8.15: AAR Visualization Navigation Menu

Monitor and analyze application usage at the enterprise level

1. From the Enterprise Detail Dashboard, the Enterprise Applications (page 512) and Data Usage on Apps per
NSG (page 512) visualizations appear automatically.
2. View the Top 20 Applications table for a detailed list of application traffic in the enterprise. Click on column
headers to sort by total bytes or to filter discovered applications.
3. Perform one of the following:
1. Click on an application in the Enterprise Applications visualization. The Data Usage per NSG for Selected
Application (page 512) visualization appears.
2. Click on an application within an NSG bar in the Data Usage on Apps per NSG visualization. The Data Usage
per NSG for Selected Application and Cumulative Data Usage on NSG for Selected Application (page 513)
visualizations appear. Go to Step 4.
4. Click on an NSG in the Data Usage per NSG for Selected Application visualization.
The Cumulative Data Usage on NSG for Selected Application visualization appears.
5. Click on a source IP in the Cumulative Data Usage on NSG for Selected Application visualization.
The NSG Application Traffic Details for Source IP (page 513) visualization appears.
6. As required, configure the Time Interval, Refresh Interval, and Traffic Type parameters to specify the recency
of the displayed data, the rate at which it is refreshed, and whether upload or download data is displayed.

Monitor and analyze application usage at the domain level

1. From the Domain Details Dashboard, the Domain Applications (page 525) and Data Usage on Apps per NSG
(page 526) visualizations appear automatically.
2. View the Top 20 Applications table for a detailed list of application traffic in the domain. Click on column
headers to sort by total bytes or packets.
3. Perform one of the following:
1. Click on an application in the Domain Applications visualization. The Data Usage per NSG for Selected Appli-
cation (page 526) visualization appears.
2. Click on an application within an NSG bar in the Data Usage on Apps per NSG visualization. The Data Usage
per NSG for Selected Application and Cumulative Data Usage on NSG for Selected Application (page 526)
visualizations appear. Go to Step 5.

8.2. AAR Visualization 509

 VNS User Guide, Release 5.3.2

4. Click on an NSG in the Data Usage per NSG for Selected Application visualization.
The Cumulative Data Usage on NSG for Selected Application visualization appears.
5. Click on a source IP in the Cumulative Data Usage on NSG for Selected Application visualization.
The NSG Application Traffic Details for Source IP (page 526) visualization appears.
6. As required, configure the Time Interval, Refresh Interval, and Traffic Type parameters to specify the recency
of the displayed data, the rate at which it is refreshed, and whether upload or download data is displayed.
7. Click on the Application Performance tab. A list of source and destination NSGs are generated automatically.
8. Select a source and destination NSG from the lists. The SLA HeatMap (page 524) visualization appears.
9. Select a box in the SLA HeatMap. A list of traffic uplink pairs is generated.
10. Select a traffic uplink pair. A graph of application traffic and three graphs tracking SLA violations are generated.

8.2.4 Scrolling Graphs

For bar graph visualizations that display more than 10 NSGs, the scrolling tool allows the user to view data outside
the scope of the visualization. Click and drag the grey box to scroll through the graph. Click and drag the edge of the
grey box to resize the box and change how many NSGs are displayed in the bar graph.

8.2. AAR Visualization 510

 VNS User Guide, Release 5.3.2

8.2.5 Enterprise Dashboard

Fig. 8.16: Enterprise Dashboard

SLA Status - Flow effectiveness score

Enterprise level representation of Application SLA status.

Computation: Possible values are In SLA, Out of SLA and/or Not monitored.
Query example: SLA Status per Enterprise (page 526)

Top 5 Discovered Applications

Enterprise level top 5 discovered applications.

Computation: Sum of total Bytes sent and/or received in descending order.
Query example: Top 5 Applications (page 527)

Top 5 Upload Users

Enterprise level top 5 upload users (IP addresses).

Computation: Sum of Ingress Bytes sourced from IP to any destination in descending order.
Query example: Top 5 Upload Users (page 528)

Top 5 Download Users

Enterprise level top 5 download users (IP addresses).

Computation: Sum of Egress Bytes for any traffic destined to this client IP in descending order.
Query example: Top 5 Download Users (page 530)

8.2. AAR Visualization 511

 VNS User Guide, Release 5.3.2

Newly Discovered Default Application

Count of L7 Classification in default application.

Computation: Distinct count of L7 Classification in default application in a given enterprise
Query example: Newly Discovered Default Application (page 531)

8.2.6 Enterprise Detail Dashboard

Fig. 8.17: Enterprise Detail Dashboard

Enterprise Applications

From this visualization, an admin user can visualize per application data usage in the enterprise. This visualization
is useful if an admin determines there is heavy traffic flowing on a particular application. The user can click on an
application in the graph to drill down and view which NSGs are generating traffic for the selected application.

Data Usage on Apps per NSG

From this visualization, an admin user can visualize total application throughput for each NSG in an enterprise. This
visualization is useful if an admin needs to see which branch is generating high amounts of traffic and whether a
particular application is responsible. The user can click on an application in the graph to drill down and view per NSG
traffic for the selected application as well as data usage on the selected NSG for the selected application.

Data Usage per NSG for Selected Application

From this visualization, an admin user can visualize per NSG traffic for the selected application in the enterprise.
This visualization is useful if an admin user needs to see which branch is generating the most traffic on a specific
application. The user can click on an NSG in the branch to view data usage on the selected NSG for the selected
application.

8.2. AAR Visualization 512

 VNS User Guide, Release 5.3.2

Cumulative Data Usage on NSG for Selected Application

From this visualization, an admin user can visualize per source IP data usage for the selected NSG and application.
This visualization is useful if an admin user needs to see if a particular client is generating most of the application
traffic. The user can click on a source IP to view total application traffic history on an NSG for the selected source IP.

NSG Application Traffic Details for Source IP

From this visualization, an admin user can visualize per application traffic flow over time on an NSG for a selected
source IP. This visualization is useful if an admin user needs to identify a spike in data usage from a particular client
on an NSG. The user can hover over a point of time in the graph to view per app usage during that time.

Fig. 8.18: NSG Application Traffic Details for Source IP

Enterprise - Top 20 Applications

Enterprise level top 20 applications.

Computation: Sum of total Bytes sent and/or received in descending order.
Query example: Top 20 Applications in Enterprise (page 532)

Application specific Date Histogram

Enterprise level discovered application total bytes usage over time.

Computation: Sum of total Bytes sent and/or received for a given interval displayed over the configured timespan.
Possible values are:
• If time span = Last 15 minutes: interval is 1 minute
• If time span = Last 24 hours: interval is 1 hour
• If period = Last 7 days: interval is 12 hours

8.2. AAR Visualization 513

 VNS User Guide, Release 5.3.2

Query example: Application Usage Histogram (page 534)

8.2.7 Enterprise Map View

The enterprise map view provides a visualization of the geographic location and administrative status of all NSGs
in the enterprise. The map view uses Google Maps and user-specified NSG locations to map branch locations on an
interactive, navigable map. Using the map view, you can search and navigate to NSGs by region and quickly determine
which NSGs are in a pre-bootstrap state or require a software upgrade.

Fig. 8.19: Enterprise Map View

Map View Prerequisites

Before the enterprise map view can be used, some mandatory configuration must be completed in the VSD. This
configuration includes specifying a Google Maps API key and ensuring that each NSG is configured with a location
using the proper address format.

Google Maps API Key

Before the enterprise map view can be used, a Google Maps API key must be provided. You must create a Google
Maps account and register the project to be provided an API key. Complete instructions can be found at the following
URL:
https://cloud.google.com/maps-platform/

Note: If a valid Google Maps API key is not provided, the enterprise map view does not load and displays an error
message.

8.2. AAR Visualization 514

 VNS User Guide, Release 5.3.2

Configuring Google Maps API Key

Note: This procedure needs to be performed only once.

Step 1 Navigate to Platform Configuration -> Settings -> System Configuration .

Step 2 Under Google Maps API Key, configure the API Key parameter with a Google Maps API key.
Step 3 Click Update.

NSG Location

Each NSG must be configured with a location before it can be plotted on the enterprise map view. The following is an
example of an NSG location using the proper format for an address in the U.S.A.:
755 Ravendale Dr,Mountain View,CA 94043,USA

Note: Prior to Release 5.3.2, the NSG location was not required in this specific format. As a result, some NSGs may
need to be updated with a valid address. The address must be updated through the VSD and not the API.

Note: If the provided address is valid, it is displayed in a small map with a location flag beneath the Address field. If
the address is invalid, this map is not displayed.

Note: A partial address can be specified for an NSG location. For example, if only a city or state name is specified as
an NSG location, it is displayed in the map view.

Note: Not all Google Maps addresses can be correctly resolved on the enterprise map view. For example, apartment
or unit numbers and certain special characters cannot be used as part of an address. If the # character is used as part of
an NSG location, it is not displayed in the map view.

Configuring NSG Location

Step 1 From the enterprise view, navigate to Infrastructure -> Network Services Gateways -> NSG
-> Bootstrapping .
Step 2 Configure the Address and Time Zone parameters.
Step 3 Click Update.
Step 4 Repeat steps 1 to 3 to configure the address and time zone for more NSGs, as required.

Map View Dashboard

The map view dashboard includes a world map and a search bar. You can use the map controls to view the map in full
screen or to zoom in or out. You can also zoom with a mouse scroll and pan by clicking and dragging.

8.2. AAR Visualization 515

 VNS User Guide, Release 5.3.2

Hover over an NSG icon to view the NSG name, address, bootstrap status, software version, and alarm counts. Click
an NSG icon to navigate to the NSG Dashboard (page 518).
Each NSG in the enterprise with a configured location is displayed on the map with an icon. If there are multiple
NSGs located in close proximity to each other, the map displays an NSG cluster. You can click on the NSG cluster to
zoom in, and click again to expand or collapse the NSGs in the cluster.

Fig. 8.20: Collapsed NSG Clusters

Fig. 8.21: Expanded NSG Cluster

Bootstrapping Status

The NSG icons indicate whether the NSG has been bootstrapped. NSGs which have been successfully bootstrapped are
displayed with a blue icon. NSGs which are inactive, revoked, or awaiting bootstrapping are displayed with a yellow

8.2. AAR Visualization 516

 VNS User Guide, Release 5.3.2

icon. You can use the NSG search bar to filter NSGs based on their bootstrapping state. This functionality allows you
to quickly identify which NSGs in the enterprise are still awaiting bootstrapping or have failed bootstrapping.

NSG Search Bar

The NSG search bar allows you to filter the NSGs to be displayed on the map. You can search based on the NSG name,
location, bootstrapping status, or software version. NSGs which do not fit the specified search criteria are hidden from
the map. The NSG search bar can be used to filter NSGs which are still awaiting bootstrapping. It can also be used to
identify which NSGs require a software upgrade.
When you place the cursor in the NSG search bar, available search criteria and operators are suggested in a drop-down
menu.
Search criteria can be specified using the following variables:
• Name — Searches on the NSG name as configured in the VSD.
• Region — Searches on the NSG city or county name as specified in the NSG location.
• bootstrapStatus — Searches on the bootstrapping statuses currently held by any NSGs on the map.
• NSGVersion — Searches on the NSG factory software version.
Search criteria can be specified using the following operators:
• == — Specify an exact string to match with the specified variable.
• != — Specify an exact string to exclude from the search.
• contains — Specify a string to be contained within the specified variable.
• !contains — Specify a contained string to exclude from the search.
• startsWith — Specify a string to begin the specified variable.
The following is an example of a search query that filters on NSGs in Paris that have not been successfully bootstrapped
yet:
Region contains Paris AND bootstrapStatus == INACTIVE

CSV Export

You can click the CSV Export button to download a .csv file with information on all NSGs in the enterprise. This
includes information on NSG location, alarm status, bootstrapping status, and UUID.

Fig. 8.22: NSG CSV Export

8.2. AAR Visualization 517

 VNS User Guide, Release 5.3.2

8.2.8 NSG Dashboard

Fig. 8.23: NSG Dashboard

NSG Application Traffic Detail

From this visualization, an admin user can visualize per application traffic flow broken down by application. This
visualization is useful if an admin user needs to view which applications are responsible for the most data usage
through an NSG. The user can click on an application to view source IP traffic history on an NSG for the selected
application.

NSG Application Traffic Details for Application

From this visualization, an admin user can visualize source IP traffic flow over time on an NSG for a selected applica-
tion. This visualization is useful if an admin user needs to identify the source of a spike in data usage from a particular
application on an NSG.
In the selected NSG has a dual uplink, the dashboard displays a per port breakdown of this visualization.

NSG Application Traffic Details for Source IP

From this visualization, an admin user can visualize per application traffic flow over time on an NSG for a selected
source IP. This visualization is useful if an admin user needs to identify a spike in data usage from a particular client
on an NSG.
In the selected NSG has a dual uplink, the dashboard displays a per port breakdown of this visualization.

8.2. AAR Visualization 518

 VNS User Guide, Release 5.3.2

Per Port Traffic Visualization

From this visualization, an admin user can visualize the realtime throughput of traffic between two NSGs for a specific
application. The visualization allows the user to visualize which path the application took at a given time. Three SLA
metrics (packet loss, latency, and jitter) are displayed to allow the user to determine the cause of a path switchover.
The colors on the graph show when the traffic flow was in SLA, out-of-SLA, or down.
The user can drill down to view the three SLA metrics overlayed on the path switchover history. This visualization is
useful to show which SLA metric violation resulted in a path switchover at a given point in time.
In the selected NSGs have a dual uplink, the dashboard displays a per port breakdown of this visualization.

Top 10 Applications

NSG level top 10 applications.

Computation: Sum of total Bytes sent and/or received in descending order.
Query example: Top 10 Applications (page 535)

Top 5 Discovered Application - Usage over Time

NSG level discovered application total bytes usage over time.

Computation: Sum of total Bytes sent and/or received for a given interval displayed over the configured time span.
Possible values are:
• If time span = Last 15 minutes: interval is 1 minute
• If time span = Last 24 hours: interval is 1 hour
• If period = Last 7 days: interval is 12 hours
Query example: Application Usage over Time (page 537)

Top 5 Talkers (Upload)

NSG level top 5 users (IP address sources) sending traffic to the network (egressing the NSG network ports).
Computation: Sum of ingress Bytes sourced from this client IP to any destination in descending order.
Query example: Top 5 Talkers (Upload) (page 538)

Top 5 Talkers (Download)

NSG level top 5 users (IP addresses) receiving traffic from the network
Computation: Sum of egress Bytes for any traffic destined to this client IP in descending order.
Query example: Top 5 Talkers (Download) (page 540)

Alarms

From this visualization, you can view a summary of the number of critical, major, and minor alarms on the NSG. You
can click the CSV Export button to download a .csv file summarizing information for the alarms on the NSG. Click
an alarm severity circle to navigate to the Alarms Dashboard (page 521).

8.2. AAR Visualization 519

 VNS User Guide, Release 5.3.2

8.2.9 NSG Detail Dashboard

Fig. 8.24: NSG Detail Dashboard

From NSG Application Report

NSG level report of applications connected to local (this) NSG sending traffic to remote NSGs.
Computation: Sum of ingress Bytes in descending order.
Query example: From NSG Application Report (page 541)

To NSG Application Report

NSG level report of applications connected to remote NSGs sending traffic to local (this) NSG.
Computation: Sum of ingress Bytes in descending order.
Query example: To NSG Application Report (page 545)

From NSG SLA Report

All SLA violation during the last time interval, that caused a path from NSG to switch.
Computation: Ordered by time in descending order.
Query example: From NSG SLA Report (page 550)

To NSG SLA Report

All SLA violation during the last time interval, that caused a path to this NSG to switch.
Computation: Ordered by time in descending order.
Query example: To NSG SLA Report (page 551)

8.2. AAR Visualization 520

 VNS User Guide, Release 5.3.2

8.2.10 NAT-T Events Dashboard

See NAT-T Visualization (page 141) for more information about the NAT-T Events dashboard.

8.2.11 IKE Dashboard

See HTTP Ping Visualization (page 564) for more information about the NAT-T Events dashboard.

8.2.12 Alarms Dashboard

The alarms dashboard displays a summary of all alarms currently raised on the NSG. From the alarms table, you can
sort on alarm severity, creation time, count, or error condition.

Fig. 8.25: Alarms Dashboard

You can click the CSV Export button to download a .csv file with information on all alarms on the NSG.

8.2.13 Domain - Applications Dashboard

Top 20 Talkers in Domain

Domain level top 10 discovered and configured applications.

Computation: Sum of total Bytes sent and/or received in descending order.
Query example: Top 20 Talkers in Domain (page 553)

Top 5 Applications in Domain

Domain level top 5 applications.

Computation: Sum of total Bytes sent and/or received in descending order.
Query example: Top 5 Applications in Domain (page 556)

8.2. AAR Visualization 521

 VNS User Guide, Release 5.3.2

Top 5 Application Groups

Domain level top 5 Application Performance Management Groups.

Computation: Sum of total Bytes sent and/or received in descending order.
Query example: Top 5 Application Groups in Domain (page 558)

8.2.14 Domain - Network Performance Dashboard

In order to view the network performance dashboard, first select the source and destination NSG in the table that is
displayed for this report (see example graphic below). NSG personality types may also be selected, in which case the
table list will display NSGs of that personality type. Once selected, the one-way path performance measurements will
be displayed in the lower panel of the same graphic for NSG as Source (page 522) and NSG as Destination (page 523).

Fig. 8.26: Domain - Network Performance Dashboard

From NSG

This table lists source NSGs in the domain. Select a source NSG to generate application performance visualizations.
Use the Personality drop-down menu to filter for NSGs, NSG-UBRs, or NSG-BRs.

To NSG

This table lists destination NSGs in the domain. Select a destination NSG to generate application performance visual-
izations. Use the Personality drop-down menu to filter for NSGs, NSG-UBRs, or NSG-BRs.

Probe Detail (NSG as Source)

Domain level local NSG (source) to remote NSG (destination) Path Performance Measurements. Displayed in the
bottom left panel of the graphic above.
Computation: Order by time in descending order.

8.2. AAR Visualization 522

 VNS User Guide, Release 5.3.2

Query example: Path Performance Measurements (Source to Destination NSG) (page 559)

Probe Detail (NSG as Destination)

Domain level local NSG (destination) to remote NSG (sources) Path Performance Measurements. Displayed in the
bottom right panel of the graphic above.
Computation: Order by time in descending order.
Query example: Path Performance Measurements (NSG as Destination) (page 560)

8.2.15 Domain - Application Performance Management Dashboard

8.2. AAR Visualization 523

 VNS User Guide, Release 5.3.2

Fig. 8.27: Domain - Application Performance Management Dashboard

From NSG

This table lists source NSGs in the domain. Select a source NSG to generate application performance visualizations.
Use the Personality drop-down menu to filter for NSGs, NSG-UBRs, or NSG-BRs.

To NSG

This table lists destination NSGs in the domain. Select a destination NSG to generate application performance visual-
izations. Use the Personality drop-down menu to filter for NSGs, NSG-UBRs, or NSG-BRs.

SLA HeatMap

The SLA HeatMap displays a historical analysis of application SLA violations for a selected pair of NSGs. The
HeatMap displays one of three colors for an application time slot:
Blue - unmonitored: the application does not have performance management enabled.
Red - out of SLA: the application has performance management enabled and the traffic flows are on a path that do not
meet SLA requirements.
Green - in SLA: the application has performance management enabled and the traffic flows are on a path that meet
SLA requirements.
You can click on any cell in the HeatMap to see a detailed view of SLA violation details and flow stats.

Traffic Uplink Pairs for Application

This table lists all uplink pairs between the selected source and destination NSGs. Select a pair of ports to generate
traffic SLA traffic data.

8.2. AAR Visualization 524

 VNS User Guide, Release 5.3.2

Traffic for Application

This graph displays the traffic throughput of the application on the selected path.

SLA Packet Loss for Application

This graph displays the packet loss percentage for the application based on the APM probes. The green line is the
application’s configured value of packet loss SLA in VSD.

SLA Delay for Application

This graph displays the latency for the application based on the APM probes. The green line is the application’s
configured value of latency SLA in VSD.

SLA Jitter for Application

This graph displays the jitter for the application based on the APM probes. The green line is the application’s config-
ured value of jitter SLA in VSD.

8.2.16 Domain - Details Dashboard

Fig. 8.28: Domain - Details Dashboard

Domain Applications

From this visualization, an admin user can visualize per application data usage in the domain. This visualization is
useful if an admin determines there is heavy traffic flowing on a particular application. The user can click on an
application in the graph to drill down and view which NSGs are generating traffic for the selected application.

8.2. AAR Visualization 525

 VNS User Guide, Release 5.3.2

Data Usage on Apps per NSG

From this visualization, an admin user can visualize per NSG traffic broken down per application in the domain. This
visualization is useful if an admin needs to see which branch is generating high amounts of traffic and whether a
particular application is responsible. The user can click on an application in the graph to drill down and view per NSG
traffic for the selected application as well as data usage on the selected NSG for the selected application.

Data Usage per NSG for Selected Application

From this visualization, an admin user can visualize per NSG traffic for the selected application in the domain. This
visualization is useful if an admin user needs to see which branch is generating the most traffic on a specific application.
The user can click on an NSG in the branch to view data usage on the selected NSG for the selected application.

Cumulative Data Usage on NSG for Selected Application

From this visualization, an admin user can visualize per source IP data usage for the selected NSG and application.
This visualization is useful if an admin user needs to see if a particular client is generating most of the application
traffic. The user can click on a source IP to view total application traffic history on an NSG for the selected source IP.

NSG Application Traffic Details for Source IP

From this visualization, an admin user can visualization per application traffic flow over time on an NSG for a selected
source IP. This visualization is useful if an admin user needs to identify a spike in data usage from a particular client
on an NSG.

8.2.17 Query Examples

Enterprise Dashboard

SLA Status - Flow effectiveness score

Enterprise level representation of Application SLA status.

Computation: Possible values are In SLA, Out of SLA and/or Not monitored.

{
"size":0,
"query":{
"bool":{
"must":[
{
"range":{
"timestamp":{
"gte":"{{startTime:now-24h}}",
"lte":"{{endTime:now}}",
"format":"epoch_millis"
}
}
}
]
}
},

8.2. AAR Visualization 526

 VNS User Guide, Release 5.3.2

"aggs":{
"1":{
"filters":{
"filters":{
"Enterprise":{
"query":{
"term":{
"EnterpriseName":"{{EnterpriseName:test_org}}"
}
}
}
}
},
"aggs":{
"slastatus":{
"terms":{
"field":"SlaStatus"
}
}
}
}
}
}

Top 5 Applications

Enterprise level top 5 discovered applications.

Computation: Sum of total Bytes sent and/or received in descending order.
{
"size":0,
"query":{
"bool":{
"must":[
{
"range":{
"timestamp":{
"gte":"{{startTime:now-24h}}",
"lte":"{{endTime:now}}",
"format":"epoch_millis"
}
}
}
]
}
},
"aggs":{
"3":{
"filters":{
"filters":{
"Enterprise":{
"query":{
"term":{
"EnterpriseName":"{{EnterpriseName:test_org}}"
}
}

8.2. AAR Visualization 527

 VNS User Guide, Release 5.3.2

}
}
},
"aggs":{
"L7Classification":{
"terms":{
"field":"L7ClassificationEnhanced",
"size":5,
"order":{
"Sum of MB":"desc"
}
},
"aggs":{
"Sum of MB":{
"sum":{
"field":"TotalBytesCount"
}
}
}
}
}
}
}
}

Top 5 Upload Users

Enterprise level top 5 upload users (IP addresses).

Computation: Sum of Ingress Bytes sourced from IP to any destination in descending order.
{
"size":0,
"query":{
"bool":{
"must":[
{
"range":{
"timestamp":{
"gte":"{{startTime:now-24h}}",
"lte":"{{endTime:now}}",
"format":"epoch_millis"
}
}
},
{
"term":{
"EgressBytes":0
}
}
]
}
},
"aggs":{
"5":{
"filters":{
"filters":{

8.2. AAR Visualization 528

 VNS User Guide, Release 5.3.2

"SourceNSG":{
"query":{
"term":{
"SourceNSG":"{{snsg:ovs-114}}"
}
}
}
}
},
"aggs":{
"SourceIP":{
"terms":{
"field":"SrcIp",
"size":5,
"order":{
"Bytes":"desc"
}
},
"aggs":{
"Packets":{
"sum":{
"field":"IngressPackets"
}
},
"Bytes":{
"sum":{
"field":"IngressBytes"
}
},
"DomainName":{
"terms":{
"field":"Domain",
"size":5,
"order":{
"Bytes":"desc"
}
},
"aggs":{
"Packets":{
"sum":{
"field":"IngressPackets"
}
},
"Bytes":{
"sum":{
"field":"IngressBytes"
}
}
}
}
}
}
}
}
}
}

8.2. AAR Visualization 529

 VNS User Guide, Release 5.3.2

Top 5 Download Users

Enterprise level top 5 download users (IP addresses).

Computation: Sum of Egress Bytes for any traffic destined to this client IP in descending order.

{
"size":0,
"query":{
"bool":{
"must":[
{
"range":{
"timestamp":{
"gte":"{{startTime:now-24h}}",
"lte":"{{endTime:now}}",
"format":"epoch_millis"
}
}
},
{
"term":{
"IngressBytes": 0
}
}
]
}
},
"aggs":{
"5":{
"filters":{
"filters":{
"DestinationNSG":{
"query":{
"term":{
"DestinationNSG":"{{snsg:ovs-114}}"
}
}
}
}
},
"aggs":{
"DestinationIP":{
"terms":{
"field":"DstIp",
"size":5,
"order":{
"Bytes":"desc"
}
},
"aggs":{
"Packets":{
"sum":{
"field":"EgressPackets"
}
},
"Bytes":{
"sum":{

8.2. AAR Visualization 530

 VNS User Guide, Release 5.3.2

"field":"EgressBytes"
}
},
"DomainName":{
"terms":{
"field":"Domain",
"size":5,
"order":{
"Bytes":"desc"
}
},
"aggs":{
"Packets":{
"sum":{
"field":"EgressPackets"
}
},
"Bytes":{
"sum":{
"field":"EgressBytes"
}
}
}
}
}
}
}
}
}
}

Newly Discovered Default Application

{
"size":0,
"query":{
"bool":{
"must":[
{
"range":{
"timestamp":{
"gte":"{{startTime:now-24h}}",
"lte":"{{endTime:now}}",
"format":"epoch_millis"
}
}
},
{
"term":{
"Application":"Default Application"
}
}
]
}
},
"aggs": {

8.2. AAR Visualization 531

 VNS User Guide, Release 5.3.2

"2": {
"filters":{
"filters":{
"Enterprise":{
"query":{
"term":{
"EnterpriseName":"{{enterpriseName:test_
˓→organization}}"

}
}
}
}
},
"aggs": {
"l7_count": {
"cardinality": {
"field":"L7ClassificationEnhanced"
}
}
}
}
}
}

Enterprise Detail Dashboard

Enterprise - Top 20 Applications

Enterprise level top 20 applications.

Computation: Sum of total Bytes sent and/or received in descending order.
..{ .. “id”:”top20-talkers-enterprise-table”, .. “title”:”Top 20 Talkers in Enterprise”, .. “service”:”elasticsearch”, ..
“query”: { .. “index”:”{{index:nuage_dpi_flowstats}}”, .. “type”:”{{type:nuage_doc_type}}”, .. “body”: {
{
"size":0,
"query":{
"bool":{
"must":[
{
"range":{
"timestamp":{
"gte":"{{startTime:now-24h}}",
"lte":"{{endTime:now}}",
"format":"epoch_millis"
}
}
}
]
}
},
"aggs":{
"2":{
"filters":{
"filters":{
"EnterpriseName":{

8.2. AAR Visualization 532

 VNS User Guide, Release 5.3.2

"query":{
"term":{
"EnterpriseName":"{{EnterpriseName:test_org}}"
}
}
}
}
},
"aggs":{
"5":{
"filters":{
"filters":{
"AppName":{
"query":{
"not":{
"term":{
"Application":"Default Application"
}
}
}
}
}
},
"aggs":{
"Application":{
"terms":{
"field":"Application",
"size":20,
"order":{
"1":"desc"
}
},
"aggs":{
"1":{
"sum":{
"field":"TotalBytesCount"
}
},
"APMGroup":{
"terms":{
"field":"APMGroup",
"size":20,
"order":{
"1":"desc"
}
},
"aggs":{
"1":{
"sum":{
"field":"TotalBytesCount"
}
},
"DomainName":{
"terms":{
"field":"Domain",
"size":20,
"order":{
"1":"desc"

8.2. AAR Visualization 533

 VNS User Guide, Release 5.3.2

}
},
"aggs":{
"1":{
"sum":{
"field":"TotalBytesCount"
}
},
"L7Classification":{
"terms":{
"field":
˓→ "L7ClassificationEnhanced",
"size":20,
"order":{
"1":"desc"
}
},
"aggs":{
"1":{
"sum":{
"field":
˓→ "TotalBytesCount"
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}

Application specific Date Histogram

Enterprise level discovered application total bytes usage over time.

Computation: Sum of total Bytes sent and/or received for a given interval displayed over the configured timespan.
Possible values are:
• If time span = Last 15 minutes: interval is 1 minute
• If time span = Last 24 hours: interval is 1 hour
• If period = Last 7 days: interval is 12 hours
{
"size":0,
"query":{
"bool":{
"must":[
{

8.2. AAR Visualization 534

 VNS User Guide, Release 5.3.2

"range":{
"timestamp":{
"gte":"{{startTime:now-24h}}",
"lte":"{{endTime:now}}",
"format":"epoch_millis"
}
}
}
]
}
},
"aggs":{
"4":{
"filters":{
"filters":{
"L7Classification":{
"query":{
"term":{
"L7Classification":"{{app}}"
}
}
}
}
},
"aggs":{
"date-histo":{
"date_histogram":{
"field":"timestamp",
"interval":"30m"
},
"aggs":{
"SumOfBytes":{
"sum":{
"field":"TotalBytesCount"
}
},
"SumOfPackets":{
"sum":{
"field":"TotalPacketsCount"
}
}
}
}
}
}
}
}

NSG Dashboard

Top 10 Applications

NSG level top 10 applications.

Computation: Sum of total Bytes sent and/or received in descending order.

8.2. AAR Visualization 535

 VNS User Guide, Release 5.3.2

{
"size": 0,
"query":{
"bool":{
"must":[
{
"range":{
"timestamp":{
"gte":"{{startTime:now-24h}}",
"lte":"{{endTime:now}}",
"format":"epoch_millis"
}
}
}
]
}
},
"aggs":{
"4":{
"filters":{
"filters":{
"Enterprise":{
"query":{
"term":{
"EnterpriseName":"{{EnterpriseName:test_organization}}"
}
}
}
}
},
"aggs":{
"3":{
"filters":{
"filters":{
"SourceNSG":{
"query":{
"term":{
"SourceNSG":"{{snsg:ovs-114}}"
}
}
}
}
},
"aggs":{
"Application":{
"terms":{
"field":"Application",
"size":10,
"order":{
"Sum of MB":"desc"
}
},
"aggs":{
"Sum of MB":{
"sum":{
"field":"TotalMB"
}
}

8.2. AAR Visualization 536

 VNS User Guide, Release 5.3.2

}
}
}
}
}
}
}
}

Top 5 Discovered Application - Usage over Time

NSG level discovered application total bytes usage over time.

Computation: Sum of total Bytes sent and/or received for a given interval displayed over the configured time span.
Possible values are:
• If time span = Last 15 minutes: interval is 1 minute
• If time span = Last 24 hours: interval is 1 hour
• If period = Last 7 days: interval is 12 hours
{
"size":0,
"query":{
"bool":{
"must":[
{
"range":{
"timestamp":{
"gte":"{{startTime:now-24h}}",
"lte":"{{endTime:now}}",
"format":"epoch_millis"
}
}
}
]
}
},
"aggs":{
"5":{
"filters":{
"filters":{
"Enterprise":{
"query":{
"term":{
"EnterpriseName":"{{EnterpriseName:test_organization}}"
}
}
}
}
},
"aggs": {
"4": {
"filters": {
"filters": {
"SourceNSG": {
"query": {

8.2. AAR Visualization 537

 VNS User Guide, Release 5.3.2

"term": {
"SourceNSG": "{{snsg:default}}"
}
}
}
}
},
"aggs": {
"ts": {
"date_histogram": {
"field": "timestamp",
"interval": "1h"
},
"aggs": {
"L7Classification": {
"terms": {
"field": "L7ClassificationEnhanced",
"size": 5,
"order": {
"SumOf": "desc"
}
},
"aggs": {
"SumOf": {
"sum": {
"field": "{{Metric:TotalBytesCount}}"
}
}
}
}
}
}
}
}
}
}
}
}

Top 5 Talkers (Upload)

NSG level top 5 users (IP addresses) sending traffic to the network.
Computation: Sum of ingress Bytes sourced from this client IP to any destination in descending order.
{
"size":0,
"query":{
"bool":{
"must":[
{
"range":{
"timestamp":{
"gte":"{{startTime:now-24h}}",
"lte":"{{endTime:now}}",
"format":"epoch_millis"
}

8.2. AAR Visualization 538

 VNS User Guide, Release 5.3.2

}
},
{
"term":{
"EgressBytes":0
}
}
]
}
},
"aggs":{
"5":{
"filters":{
"filters":{
"SourceNSG":{
"query":{
"term":{
"SourceNSG":"{{snsg:ovs-114}}"
}
}
}
}
},
"aggs":{
"SourceIP":{
"terms":{
"field":"SrcIp",
"size":5,
"order":{
"Bytes":"desc"
}
},
"aggs":{
"Packets":{
"sum":{
"field":"IngressPackets"
}
},
"Bytes":{
"sum":{
"field":"IngressBytes"
}
},
"DomainName":{
"terms":{
"field":"Domain",
"size":5,
"order":{
"Bytes":"desc"
}
},
"aggs":{
"Packets":{
"sum":{
"field":"IngressPackets"
}
},
"Bytes":{

8.2. AAR Visualization 539

 VNS User Guide, Release 5.3.2

"sum":{
"field":"IngressBytes"
}
}
}
}
}
}
}
}
}
}

Top 5 Talkers (download)

NSG level top 5 users (IP addresses) receiving traffic from the network
Computation: Sum of egress Bytes for any traffic destined to this client IP in descending order.
{
"size":0,
"query":{
"bool":{
"must":[
{
"range":{
"timestamp":{
"gte":"{{startTime:now-24h}}",
"lte":"{{endTime:now}}",
"format":"epoch_millis"
}
}
},
{
"term":{
"IngressBytes": 0
}
}
]
}
},
"aggs":{
"5":{
"filters":{
"filters":{
"DestinationNSG":{
"query":{
"term":{
"DestinationNSG":"{{snsg:ovs-114}}"
}
}
}
}
},
"aggs":{
"DestinationIP":{
"terms":{

8.2. AAR Visualization 540

 VNS User Guide, Release 5.3.2

"field":"DstIp",
"size":5,
"order":{
"Bytes":"desc"
}
},
"aggs":{
"Packets":{
"sum":{
"field":"EgressPackets"
}
},
"Bytes":{
"sum":{
"field":"EgressBytes"
}
},
"DomainName":{
"terms":{
"field":"Domain",
"size":5,
"order":{
"Bytes":"desc"
}
},
"aggs":{
"Packets":{
"sum":{
"field":"EgressPackets"
}
},
"Bytes":{
"sum":{
"field":"EgressBytes"
}
}
}
}
}
}
}
}
}
}

NSG Detail Dashboard

From NSG Application Report

NSG level report of applications connected to local (this) NSG sending traffic to remote NSGs.
Computation: Sum of ingress Bytes in descending order.

{
"size":0,
"query":{

8.2. AAR Visualization 541

 VNS User Guide, Release 5.3.2

"bool":{
"must":[
{
"range":{
"timestamp":{
"gte":"{{startTime:now-24h}}",
"lte":"{{endTimeTime:now}}",
"format":"epoch_millis"
}
}
}
]
}
},
"aggs":{
"12":{
"filters":{
"filters":{
"Enterprise":{
"query":{
"term":{
"EnterpriseName":"{{EnterpriseName:test_organization}}"
}
}
}
}
},
"aggs": {
"11": {
"filters": {
"filters": {
"SourceNSG": {
"query": {
"term": {
"SourceNSG": "{{snsg:default}}"
}
}
}
}
},
"aggs": {
"SumofBytes": {
"sum": {
"field": "TotalBytesCount"
}
},
"SumofPackets": {
"sum": {
"field": "TotalPacketsCount"
}
},
"APMGroup": {
"terms": {
"field": "APMGroup",
"size": 5,
"order": {
"SumofBytes": "desc"
}

8.2. AAR Visualization 542

 VNS User Guide, Release 5.3.2

},
"aggs": {
"SumofBytes": {
"sum": {
"field": "TotalBytesCount"
}
},
"SumofPackets": {
"sum": {
"field": "TotalPacketsCount"
}
},
"Application": {
"terms": {
"field": "Application",
"size": 5,
"order": {
"SumofBytes": "desc"
}
},
"aggs": {
"SumofBytes": {
"sum": {
"field": "TotalBytesCount"
}
},
"SumofPackets": {
"sum": {
"field": "TotalPacketsCount"
}
},
"L7Classification": {
"terms": {
"field": "L7ClassificationEnhanced",
"size": 5,
"order": {
"SumofBytes": "desc"
}
},
"aggs": {
"SumofBytes": {
"sum": {
"field": "TotalBytesCount"
}
},
"SumofPackets": {
"sum": {
"field": "TotalPacketsCount"
}
},
"SrcVportName": {
"terms": {
"field": "SrcVportName",
"size": 5,
"order": {
"SumofBytes": "desc"
}
},

8.2. AAR Visualization 543

 VNS User Guide, Release 5.3.2

"aggs": {
"SumofBytes": {
"sum": {
"field":
˓→ "TotalBytesCount"
}
},
"SumofPackets": {
"sum": {
"field":
˓→ "TotalPacketsCount"
}
},
"SrcUplink": {
"terms": {
"field": "SrcUplink",
"size": 5,
"order": {
"SumofBytes": "desc
˓→ "
}
},
"aggs": {
"SumofBytes": {
"sum": {
"field":
˓→ "TotalBytesCount"
}
},
"SumofPackets": {
"sum": {
"field":
˓→ "TotalPacketsCount"
}
},
"SrcUplinkRole": {
"terms": {
"field":
˓→ "SrcUplinkRole",
"size": 5,
"order": {
"SumofBytes
˓→ ": "desc"
}
},
"aggs": {
"SumofBytes": {
"sum": {
"field
˓→ ": "TotalBytesCount"
}
},
"SumofPackets":
˓→ {
"sum": {
"field
˓→ ": "TotalPacketsCount"
}

8.2. AAR Visualization 544

 VNS User Guide, Release 5.3.2

},
"DestinationNSG
˓→ ": {
"terms": {
"field
˓→ ": "DestinationNSG",
"size":
˓→ 5,
"order
˓→ ": {

˓→ "SumofBytes": "desc"
}
},
"aggs": {

˓→ "SumofBytes": {

˓→ "sum": {

˓→ "field": "TotalBytesCount"
}
},

˓→ "SumofPackets": {

˓→ "sum": {

˓→ "field": "TotalPacketsCount"
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}

To NSG Application Report

NSG level report of applications connected to remote NSGs sending traffic to local (this) NSG.
Computation: Sum of ingress Bytes in descending order.

8.2. AAR Visualization 545

 VNS User Guide, Release 5.3.2

{
"size":0,
"query":{
"bool":{
"must":[
{
"range":{
"timestamp":{
"gte":"{{startTime:now-24h}}",
"lte":"{{endTimeTime:now}}",
"format":"epoch_millis"
}
}
}
]
}
},
"aggs":{
"12":{
"filters":{
"filters":{
"Enterprise":{
"query":{
"term":{
"EnterpriseName":"{{EnterpriseName:test_organization}}"
}
}
}
}
},
"aggs": {
"11": {
"filters": {
"filters": {
"DestinationNSG": {
"query": {
"term": {
"DestinationNSG": "{{snsg:default}}"
}
}
}
}
},
"aggs": {
"SumofBytes": {
"sum": {
"field": "TotalBytesCount"
}
},
"SumofPackets": {
"sum": {
"field": "TotalPacketsCount"
}
},
"APMGroup": {
"terms": {
"field": "APMGroup",
"size": 5,

8.2. AAR Visualization 546

 VNS User Guide, Release 5.3.2

"order": {
"SumofBytes": "desc"
}
},
"aggs": {
"SumofBytes": {
"sum": {
"field": "TotalBytesCount"
}
},
"SumofPackets": {
"sum": {
"field": "TotalPacketsCount"
}
},
"Application": {
"terms": {
"field": "Application",
"size": 5,
"order": {
"SumofBytes": "desc"
}
},
"aggs": {
"SumofBytes": {
"sum": {
"field": "TotalBytesCount"
}
},
"SumofPackets": {
"sum": {
"field": "TotalPacketsCount"
}
},
"L7Classification": {
"terms": {
"field": "L7ClassificationEnhanced",
"size": 5,
"order": {
"SumofBytes": "desc"
}
},
"aggs": {
"SumofBytes": {
"sum": {
"field": "TotalBytesCount"
}
},
"SumofPackets": {
"sum": {
"field": "TotalPacketsCount"
}
},
"SrcVportName": {
"terms": {
"field": "SrcVportName",
"size": 5,
"order": {

8.2. AAR Visualization 547

 VNS User Guide, Release 5.3.2

"SumofBytes": "desc"
}
},
"aggs": {
"SumofBytes": {
"sum": {
"field":
˓→ "TotalBytesCount"
}
},
"SumofPackets": {
"sum": {
"field":
˓→ "TotalPacketsCount"
}
},
"SrcUplink": {
"terms": {
"field": "SrcUplink",
"size": 5,
"order": {
"SumofBytes": "desc
˓→ "
}
},
"aggs": {
"SumofBytes": {
"sum": {
"field":
˓→ "TotalBytesCount"
}
},
"SumofPackets": {
"sum": {
"field":
˓→ "TotalPacketsCount"
}
},
"SrcUplinkRole": {
"terms": {
"field":
˓→ "SrcUplinkRole",
"size": 5,
"order": {
"SumofBytes
˓→ ": "desc"
}
},
"aggs": {
"SumofBytes": {
"sum": {
"field
˓→ ": "TotalBytesCount"
}
},
"SumofPackets":
˓→ {
"sum": {

8.2. AAR Visualization 548

 VNS User Guide, Release 5.3.2

"field
˓→ ": "TotalPacketsCount"
}
},
"SourceNSG": {
"terms": {
"field
˓→ ": "SourceNSG",
"size":
˓→ 5,
"order
˓→ ": {

˓→ "SumofBytes": "desc"
}
},
"aggs": {

˓→ "SumofBytes": {

˓→ "sum": {

˓→ "field": "TotalBytesCount"
}
},

˓→ "SumofPackets": {

˓→ "sum": {

˓→ "field": "TotalPacketsCount"
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}

8.2. AAR Visualization 549

 VNS User Guide, Release 5.3.2

From NSG SLA Report

NSG level report of applications connected to local (this) NSG and their SLA status. Possible values are In SLA, Out
of SLA and/or Not monitored.
Computation: Ordered by time in descending order.

{
"size":0,
"query":{
"bool":{
"must":[
{
"range":{
"timestamp":{
"gte":"{{startTime:now-24h}}",
"lte":"{{endTimeTime:now}}",
"format":"epoch_millis"
}
}
}
]
}
},
"aggs": {
"11": {
"filters": {
"filters": {
"SourceNSG": {
"query": {
"term": {
"SourceNSG": "{{snsg:default}}"
}
}
}
}
},
"aggs": {
"ts": {
"terms": {
"field": "timestamp",
"size": 10,
"order": {
"_count": "desc"
}
},
"aggs": {
"DestinationNSG": {
"terms": {
"field": "DestinationNSG",
"size": 10,
"order": {
"_count": "desc"
}
},
"aggs": {
"Application": {
"terms": {

8.2. AAR Visualization 550

 VNS User Guide, Release 5.3.2

"field": "Application",
"size": 10,
"order": {
"_count": "desc"
}
},
"aggs": {
"APMGroup": {
"terms": {
"field": "APMGroup",
"size": 10,
"order": {
"_count": "desc"
}
},
"aggs": {
"ViolationType": {
"terms": {
"field": "ViolationType",
"size": 10,
"order": {
"_count": "desc"
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}

To NSG SLA Report

To NSG SLA report. NSG level report of applications connected to remote NSGs sending traffic to local (this) NSG.
Possible values are In SLA, Out of SLA and/or Not monitored.
Computation: Ordered by time in descending order.

{
"size":0,
"query":{
"bool":{
"must":[
{
"range":{
"timestamp":{
"gte":"{{startTime:now-24h}}",
"lte":"{{endTimeTime:now}}",
"format":"epoch_millis"

8.2. AAR Visualization 551

 VNS User Guide, Release 5.3.2

}
}
}
]
}
},
"aggs": {
"11": {
"filters": {
"filters": {
"DestinationNSG": {
"query": {
"term": {
"DestinationNSG": "{{snsg:default}}"
}
}
}
}
},
"aggs": {
"ts": {
"terms": {
"field": "timestamp",
"size": 10,
"order": {
"_count": "desc"
}
},
"aggs": {
"SourceNSG": {
"terms": {
"field": "SourceNSG",
"size": 10,
"order": {
"_count": "desc"
}
},
"aggs": {
"Application": {
"terms": {
"field": "Application",
"size": 10,
"order": {
"_count": "desc"
}
},
"aggs": {
"APMGroup": {
"terms": {
"field": "APMGroup",
"size": 10,
"order": {
"_count": "desc"
}
},
"aggs": {
"ViolationType": {
"terms": {

8.2. AAR Visualization 552

 VNS User Guide, Release 5.3.2

"field": "ViolationType",
"size": 10,
"order": {
"_count": "desc"
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}

Domain - Applications Dashboard

Top 20 Talkers in Domain

Domain level top 10 discovered and configured applications.

Computation: Sum of total Bytes sent and/or received in descending order.

{
"size": 0,
"query": {
"bool": {
"must": [
{
"range": {
"timestamp": {
"gte": "{{startTime:now-24h}}",
"lte": "{{endTime:now}}",
"format": "epoch_millis"
}
}
}
]
}
},
"aggs": {
"2": {
"filters": {
"filters": {
"Enterprise": {
"query": {
"term": {
"EnterpriseName": "{{EnterpriseName:test_org}}"
}
}
}

8.2. AAR Visualization 553

 VNS User Guide, Release 5.3.2

}
},
"aggs": {
"3": {
"filters": {
"filters": {
"Domain": {
"query": {
"term": {
"Domain": "{{domainName:Domain1}}"
}
}
}
}
},
"aggs": {
"Application": {
"terms": {
"field": "Application",
"size": 20,
"order": {
"1": "desc"
}
},
"aggs": {
"1": {
"sum": {
"field": "TotalBytesCount"
}
},
"11": {
"sum": {
"field": "TotalPacketsCount"
}
},
"L7Classification": {
"terms": {
"field": "L7ClassificationEnhanced",
"size": 20,
"order": {
"1": "desc"
}
},
"aggs": {
"1": {
"sum": {
"field": "TotalBytesCount"
}
},
"11": {
"sum": {
"field": "TotalPacketsCount"
}
},
"SrcVportName": {
"terms": {
"field": "SrcVportName",
"size": 20,

8.2. AAR Visualization 554

 VNS User Guide, Release 5.3.2

"order": {
"1": "desc"
}
},
"aggs": {
"1": {
"sum": {
"field": "TotalBytesCount"
}
},
"11": {
"sum": {
"field": "TotalPacketsCount"
}
},
"SourceNSG": {
"terms": {
"field": "SourceNSG",
"size": 20,
"order": {
"1": "desc"
}
},
"aggs": {
"1": {
"sum": {
"field":
˓→"TotalBytesCount"
}
},
"11":{
"sum": {
"field":
˓→"TotalPacketsCount"
}
},
"DestinationNSG": {
"terms": {
"field":
˓→"DestinationNSG",
"size": 20,
"order": {
"1": "desc"
}
},
"aggs": {
"1": {
"sum": {
"field":
˓→"TotalBytesCount"
}
},
"11": {
"sum": {
"field":
˓→"TotalPacketsCount"
}
},

8.2. AAR Visualization 555

 VNS User Guide, Release 5.3.2

"DestVportName": {
"terms": {
"field":
˓→ "DestVportName",
"size": 20,
"order": {
"1": "desc"
}
},
"aggs": {
"1": {
"sum": {
"field
˓→ ": "TotalBytesCount"
}
},
"11": {
"sum": {
"field
˓→ ": "TotalPacketsCount"
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}

Top 5 Applications

Domain level top 5 applications.

Computation: Sum of total Bytes sent and/or received in descending order.
{
"size":0,
"query":{
"bool":{
"must":[
{
"range":{
"timestamp":{
"gte":"{{startTime:now-24h}}",
"lte":"{{endTime:now}}",

8.2. AAR Visualization 556

 VNS User Guide, Release 5.3.2

"format":"epoch_millis"
}
}
}
]
}
},
"aggs":{
"3":{
"filters":{
"filters":{
"Enterprise":{
"query":{
"term":{
"EnterpriseName":"{{EnterpriseName:test_org}}"
}
}
}
}
},
"aggs":{
"3":{
"filters":{
"filters":{
"Domain":{
"query":{
"term":{
"Domain":"{{domainName:Domain1}}"
}
}
}
}
},
"aggs":{
"Application":{
"terms":{
"field":"Application",
"size":5,
"order":{
"SumofBytes":"desc"
}
},
"aggs":{
"SumofBytes":{
"sum":{
"field":"TotalBytesCount"
}
}
}
}
}
}
}
}
}
}

8.2. AAR Visualization 557

 VNS User Guide, Release 5.3.2

Top 5 Application Groups

Domain level top 5 Application Performance Management Groups.

Computation: Sum of total Bytes sent and/or received in descending order.

{
"size":0,
"query":{
"bool":{
"must":[
{
"range":{
"timestamp":{
"gte":"{{startTime:now-24h}}",
"lte":"{{endTimeTime:now}}",
"format":"epoch_millis"
}
}
}
]
}
},
"aggs":{
"3":{
"filters":{
"filters":{
"Enterprise":{
"query":{
"term":{
"EnterpriseName":"{{EnterpriseName:test_org}}"
}
}
}
}
},
"aggs":{
"4":{
"filters":{
"filters":{
"Domain":{
"query":{
"term":{
"Domain":"{{domainName:Domain1}}"
}
}
}
}
},
"aggs":{
"Sum of MB":{
"sum":{
"field":"TotalMB"
}
},
"APMGroup":{
"terms":{
"field":"APMGroup",

8.2. AAR Visualization 558

 VNS User Guide, Release 5.3.2

"size":5,
"order":{
"Sum of MB":"desc"
}
},
"aggs":{
"Sum of MB":{
"sum":{
"field":"TotalMB"
}
}
}
}
}
}
}
}
}
}

Domain - Network Performance Dashboard

Probe Detail (NSG as Source)

Domain level local NSG (source) to remote NSG (destination) Path Performance Measurements.
Computation: Order by time in descending order.

{
"sort": [
{ "timestamp": { "order": "desc" } }
],
"query": {
"bool": {
"should": [
{
"bool": {
"must": [
{
"term":
{"SourceNSG": "{{snsg}}" }
},
{ "term":
{"DestinationNSG": "{{dnsg}}"}
},
{ "range": {
"timestamp": {
"gte": "{{startTime:now-24h}}",
"lte": "{{endTime:now}}",
"format":"epoch_millis"
}
}
}
]
}
}

8.2. AAR Visualization 559

 VNS User Guide, Release 5.3.2

]
}
}
}

Probe Detail (NSG as Destination)

Domain level local NSG (destination) to remote NSG (sources) Path Performance Measurements.
Computation: Order by time in descending order.

{
"sort": [
{ "timestamp": { "order": "desc" } }
],
"query": {
"bool": {
"should": [
{
"bool": {
"must": [
{
"term":
{"SourceNSG": "{{dnsg}}"}
},
{
"term": {"DestinationNSG": "{{snsg}}"}
},
{ "range": {
"timestamp": {
"gte": "{{startTime:now-24h}}",
"lte": "{{endTime:now}}",
"format":"epoch_millis"
}
}
}
]
}
}
]
}
}
}

8.2. AAR Visualization 560

 VNS User Guide, Release 5.3.2

8.3 SaaS Routing

• SaaS routing overview (page 561)

– Supported SaaS routing features (page 561)
• SaaS application discovery (page 562)
• SaaS performance and reachability monitoring (page 562)
– HTTP ping performance monitoring with CLI (page 563)
– HTTP Ping Visualization (page 564)
• SaaS path selection (page 566)
– Per uplink NAT (page 566)
– Per uplink breakout preference (page 567)
• SaaS routing configuration specifics (page 567)
– SaaS application types (page 567)
– SaaS application groups (page 568)
– HTTP performance monitor (page 568)
– Forwarding path lists (page 569)
• Procedures for SaaS routing (page 570)
– Configuring SaaS application types (page 570)
– Configuring SaaS application groups (page 570)
– Creating a Performance Monitor for HTTP Ping (page 570)
– Assigning an HTTP performance monitor to an IKE tunnel (page 571)
– Configuring forwarding path lists (page 571)
• Predefined SaaS application type updates (page 572)
– Predefined SaaS application type update sample (page 572)
• SaaS routing CLI samples (page 572)

8.3.1 SaaS routing overview

Nuage VNS supports performance-based routing for Software as a Service (SaaS) applications such as Github or
WebEx. This feature set supports first-packet application discovery, performance and reachability monitoring, and
performance-based path selection with failover mechanisms. The SaaS feature set provides similar functionality to
AAR, while adjusting for the performance requirements and specific use cases associated with SaaS applications.

Supported SaaS routing features

Nuage VNS currently provides support for the following SaaS routing features:
• SaaS application discovery (page 562)

8.3. SaaS Routing 561

 VNS User Guide, Release 5.3.2

• SaaS performance and reachability monitoring (page 562)

• SaaS path selection (page 566)

8.3.2 SaaS application discovery

SaaS application discovery employs first packet discovery to ensure that when applying application specific policies,
the traffic flow does not switch paths after the application is discovered. The VSD maintains a database of SaaS
network macros with associated IP addresses for predefined and custom SaaS applications. Destination addresses for
traffic flows routing through an NSG are matched against the IPs in this database to identify SaaS application traffic
from the first packet.
The VSD supports a predefined list of network macros for the following SaaS application types:
• Amazon AWS
• Azure
• Github
• Google
• Jira
• Office 365
• Salesforce
• WebEx
The VSD provides an API to import a .json file with updates to the list of network macros for the predefined SaaS
application types. Network macros cannot be added to the predefined SaaS application types manually through the
VSD UI; they can be updated only with the .json file import. See Predefined SaaS application type updates (page 572)
for more information about making updates to the network macro list for predefined SaaS application types.
You can also create custom SaaS application types with manually defined network macros. These SaaS application
types cannot be updated with the .json file import, and must be maintained by the user.
SaaS application types can be associated using SaaS application groups. These SaaS application groups can then be
used as destinations in policy rules for ingress forwarding policies and ingress security policies. These policies allow
you to specify an uplink preference, specify a QoS forwarding class, route an application to a cloud security provider
via an IKE tunnel, or configure local Internet breakout for specific applications while forwarding other applications to
the overlay.

8.3.3 SaaS performance and reachability monitoring

SaaS routing performance and reachability monitoring is achieved using HTTP ping performance monitors by probing
user-defined HTTP(S) endpoints.
You can configure a performance monitor to perform an HTTP ping. The HTTP performance monitor can be used
to determine whether an active IKE tunnel has failed and to perform a switchover to a standby tunnel, if required.
These performance monitors are similar to the one-way active measure protocol probes used for AAR, but they do not
require an NSG as the destination target.
The performance monitor sends probe packets to destination target URLs at specified intervals. If the configured probe
timeout is exceeded without an HTTP response, or if the response is a failure type such as 404 Page not found or 403
Forbidden, the probe is considered unsuccessful. After the number of consecutive unsuccessful probes exceeds the
configured threshold, the VSD declares the tunnel down and raises an alarm.

8.3. SaaS Routing 562

 VNS User Guide, Release 5.3.2

When you configure an HTTP performance monitor, you can define URL destination targets for two different tiers.
Tier 1 allows you to specify a down threshold count and URL weight for each destination URL in the tier. Critical
destination URLs can be allocated a high URL weight so that failure to reach these targets is more likely to result in the
tunnel being considered down. This tier is intended for a small number of URLs that should be probed individually.
Tier 2 allows you to specify a down threshold count at the tier level for all configured destination URLs. This tier is
intended for a large number of destination URLs with the goal of providing an indication of whether the IKE tunnel
has consistent reachability to the Internet. Tier 2 is more likely to provide an accurate indication if there are more
destination URLs configured.
Defining multiple destination target URLs in tiers 1 and 2 allow you to probe the reachability and delay of multiple
endpoints at once. This provides a general indication of Internet or SaaS service cluster connectivity with greater
weight given to critical destination target URLs.
For each tier, you can specify HTTP or HTTPS destination URLs using either GET or HEAD HTTP request methods.
In a dual uplink scenario, you can assign an HTTP performance monitor to an IKE tunnel on each uplink. Probes
monitor Internet reachability on each uplink and use switchover logic to determine if traffic should be rerouted to
the standby tunnel. If an IKE tunnel on one of the uplinks does not have a performance monitor assigned to it, it is
presumed to be up. An IKE tunnel with an HTTP monitor in the up state is preferred over an IKE tunnel without an
HTTP monitor, which is in turn preferred over an IKE tunnel with an HTTP monitor in the down state.

Note: For each tier, the HEAD HTTP request method is recommended.

Note: QoS is not supported for HTTP ping. All probes are sent using the best-effort service class H.

Note: The HTTP ping feature does not support URL redirection when web servers respond with a 3xx error code.
In this case, the feature treats the ping attempt as a failure and adds a failure_reason. Use the ovs-appctl -t
nuage_perfd httping/dump-probe-stats command to view the failure_reason. As a workaround, manu-
ally update the failing URL with the web server’s redirected URL.

Note: The HTTP ping feature expects HTTP/1.1 web servers to honor HTTP requests with the absoluteURI format
in the Request Line. For example, GET http://gateway.zscalerbeta.net/vpntest HTTP/1.1. If an
HTTP/1.1 web server does not honor the absoluteURI format in the HTTP request, it may respond with a “404 Not
Found” error, which the HTTP ping feature treats as a failure.

HTTP ping performance monitoring with CLI

You can use the ovs-appctl -t nuage-probed httping/dump show command to view the down count
(Down cnt) and success count (Success cnt) parameters which increment when there are consecutive failed or suc-
cessful probes.
You can use the ovs-appctl -t nuage-probed httping/dump-probe-stats show command to view
the failure_reason of failed HTTP ping attempts. The following are possible values for the failure_reason:
• DNS resolution for URL failed
• TCP connection to URL failed
• Probe request timed out
• Send probe request failed

8.3. SaaS Routing 563

 VNS User Guide, Release 5.3.2

• Server returned xxx error, for example Server returned ‘302 Moved Temporaraily
• Read HTTP response failed
• Failed to queue the DNS request for URL
See CLI Commands (page 572) for more information about CLI commands available for HTTP performance monitors.

HTTP Ping Visualization

HTTP ping visualizations are available on the visualization dashboard and can be used to monitor and historically
analyze performance monitors of type HTTP Ping that are associated with IKE tunnels on NSG uplinks. See the
Application-Aware Routing Visualization (page 505) chapter for more information about accessing the visualization
feature set.
HTTP ping visualization can be viewed from the IKE dashboard for the selected NSG. The Tunnel Status for all
Tunnels on NSG visualization displays a heat map of chronological IKE tunnel reachability. Four different colors are
displayed in the heat map:
• Green: IKE tunnel is up
• Red: IKE tunnel is down
• Yellow: IKE tunnel is unstable with multiple state changes
• White: No data

The Traffic Bytes visualization displays the chronological progression of throughput on each tunnel. You can select
Tx bytes/packets or Rx bytes/packets to be displayed from the drop-down menu. Hovering over a point on the line
graph displays more information for that point in time. You can view the tunnel name, total bytes/packets sent in that
interval, and total throughput in bytes/packets per second.

8.3. SaaS Routing 564

 VNS User Guide, Release 5.3.2

The Probe Status for all Tunnels on NSG visualization displays a heat map of chronological tunnel probe reachability.
If the tunnel is configured with an HTTP ping probe, the status of the ping results can be monitored with the heat map.
Five different colors are displayed in the heat map:
• Green: Probe is up
• Red: Probe is down
• Grey: Probing has stopped because the IKE tunnel is down
• Yellow: Probe is unstable with multiple state changes
• Blue: Probe is in a hold down state
• White: No data

You can hover over a square in the heat map for the Probe Status for all Tunnels on NSG visualization to view the
timestamp and the detailed monitoring information of individual URLs in tiers 1 and 2 of the selected probe during the
selected interval. You can click on the square to populate the Last 8 IKE Probe Status Events visualization. The status
events visualizations list the last eight probe status change events that occurred during or before the selected interval.
For example, if the selected interval is 11:56 - 11:58, the status events table displays the last 8 events that occurred
before 11:58. The Flapping field indicates whether there were multiple state changes within the reporting interval,
suggesting that the reported status may be unreliable.

You can select a status event in the Last 8 IKE Probe Status Events table to show the detailed Tunnel Tier1 URL events
and Tunnel Tier2 URL events in the tables below. These visualizations list the results of probe events related to specific
URLs during the selected reporting interval. If a failure reason was reported, it is listed in the table.

8.3. SaaS Routing 565

 VNS User Guide, Release 5.3.2

Note: Probes and tunnels are identified by their names. If a probe name or tunnel name is changed after being
discovered by the IKE visualization, the object is treated as a new probe or tunnel in the visualization outputs.

You can configure the Time Interval and Refresh Interval parameters to specify the recency of the displayed data
and the rate at which it is refreshed.

8.3.4 SaaS path selection

SaaS path selection includes multiple options for Internet breakout, including overlay, route to underlay, PAT to under-
lay, and IKE from underlay. This functionality includes failover mechanisms to allow SaaS application traffic to switch
to a standby path if the primary path falls below performance standards. The path selection functionality provides the
option to assign multiple breakout options to different uplinks on the same NSG. This means that the failover standby
path can be a different breakout type from the primary path.

Per uplink NAT

Underlay routing and NAT can be enabled at the uplink level. In a dual uplink scenario, this means that one uplink
can be configured for underlay routing to the MPLS WAN and one uplink can be configured for underlay NAT to the
Internet.
The underlay routing and underlay NAT settings at the domain, zone, and subnet level take precedence over the settings
at the uplink level. This means that traffic will not be routed to the underlay or sent via underlay NAT unless underlay
routing or underlay NAT is also enabled at the domain, zone, or subnet level. If underlay routing and underlay NAT
are enabled on both the uplink and the domain, zone, or subnet, underlay NAT is preferred.
Per uplink underlay routing and NAT can be enabled by configuring the Underlay Routing and Underlay NAT
parameters on the uplink connection configuration form. See Define Uplink Connections (page 54) for procedural
information on configuring uplink connections.

Note: A reload config is required after configuring the Underlay Routing and Underlay NAT parameters.

Note: Per uplink NAT is not supported for NSG-BRs or NSG-UBRs.

The following table describes examples of per uplink PAT routing behavior. In these examples, underlay routing and
underlay NAT are enabled at the domain level.

8.3. SaaS Routing 566

 VNS User Guide, Release 5.3.2

Domain setting
Behavior Example
Rout- NAT
ing
En- Not Route to underlay only. Secondary-over-primary uplink preference. If secondary uplink is
abled en- Honor uplink preference. enabled for underlay routing, route to underlay on the secondary
abled uplink. Else, if primary uplink is enabled for underlay routing, route
to underlay on the primary uplink. Else, drop traffic.
En- En- NAT to underlay based on Secondary-over-primary uplink preference. If secondary uplink is
abled abled uplink capability. Honor enabled for underlay NAT, NAT to underlay on the secondary upink.
uplink preference. Else, if primary uplink is enabled for underlay routing or underlay
NAT, route or NAT to underlay on the primary uplink. Else, drop
traffic.

Per uplink breakout preference

Forwarding path lists allow you to assign an uplink preference for different breakout types. You can specify that traffic
be redirected through an IKE tunnel, or through PAT to underlay and route to underlay. If you specify the underlay
path, you can assign primary and secondary uplink preferences for PAT to underlay and route to underlay. Forwarding
path lists are applicable to the per uplink NAT feature which enables different breakout types for uplinks on the same
NSG. Together, these mechanisms allow you to configure an NSG with one MPLS link and one Internet link.
Forwarding path lists can be assigned as an action for ingress forwarding policies. Refer to the VSP User Guide for
more information about ingress forwarding policies.

8.3.5 SaaS routing configuration specifics

The following VSD objects and policies are relevant to SaaS routing.

SaaS application types

SaaS application types define a list of network macros with associated IPs for a particular SaaS application. In addition
to the eight predefined SaaS application types, you can create custom ones. For each custom SaaS application types,
you can manually define network macros with associated IPs of SaaS applications.
Network macros are organization-wide defined macros that can be used as a destination of a policy rule. For example,
a network macro could represent internal intranet access and could be used as a destination for a policy rule to drop
any packet that is coming from a particular port.
Navigate to Applications -> SaaS Application Tpes.
The following parameters can be configured for SaaS application types:
• Name
• Description
The following parameters can be configured for network macros:
• Name
• Network: Specifies either IPv4 or IPv6.
• Network IPv4 / Network IPv6: Specifies the IP address for the SaaS application.

8.3. SaaS Routing 567

 VNS User Guide, Release 5.3.2

SaaS application groups

SaaS application groups allow you to associate multiple SaaS application types. These SaaS application groups can
then be used as destinations in policy rules for ingress forwarding policies and ingress security policies. Specify SaaS
Application Group for the Destination and then select a SaaS application group from the drop-down menu.
Refer to the VSP User Guide for more information about creating policy rules for ingress forwarding policies and
ingress security policies.
Navigate to Applications -> SaaS Application Types -> SaaS Application Groups.
The following parameters can be configured for SaaS application groups:
• Name
• Description

Note: SaaS application types must be in a SaaS application group before they can be assigned to an ACL policy, even
if there is only one SaaS application type in the group.

HTTP performance monitor

HTTP performance monitors are used to assess IKE tunnel reachability and determine whether a switchover to a
standby tunnel is required. Each HTTP performance monitor includes two tiers for which you can define URL targets.
These HTTP performance monitors can be associated with an IKE tunnel.

From the enterprise view, navigate to Applications -> Performance Monitors .

The following parameters can be configured for performance monitors:
• Name
• Description
• Hold Down Timer (ms): Defines how long the VSD waits for a response before considering the performance
monitor unsuccessful.
• Probe Type: Select HTTP.
Parameters for tier 1 are configured at the URL level only. The following parameters can be configured for Tier 1 URL
targets:
• Down Threshold Count: Defines the number of consecutive failed probes before the VSD declares a state
change and raises an alarm.
• HTTP Request Type: Specifies the HTTP request method used for the HTTP ping.
• URL Weight in %: Defines the percent weight for the URL within Tier 1. The sum of the URL weights in tier
1 cannot exceed 100.
• URL: Defines the HTTP or HTTPS target. Specify one URL up to 2000 characters long.
• Packet Count:: Defines the number of packets sent at the specified interval.
• Probe Interval: Defines the rate at which probe packets are sent to the destination target.
• Timeout (in ms): Defines how long the VSD waits for a response before considering a probe unsuccessful.
Parameters for tier 2 are configured globally at the tier level as well as locally at the URL level. The following
parameters can be configured globally for Tier 2:

8.3. SaaS Routing 568

 VNS User Guide, Release 5.3.2

• Down Threshold Count: Defines the number of consecutive failed probes before the VSD declares a state
change and raises an alarm.
• Packet Count:: Defines the number of packets sent at the specified interval.
• Probe Interval: Defines the rate at which probe packets are sent to the destination target.
• Timeout (in ms): Defines how long the VSD waits for a response before considering a probe unsuccessful.

Note: The Down Threshold Count parameter must be less than or equal to the number of destination URLs config-
ured for Tier 2.

Note: The default value for the Probe Interval is 1 packet every 10 seconds. Setting the value to a more frequent
interval, such as 10 packets every 1 second, can result in inconsistent failures.

The following parameters can be configured for Tier 2 URL targets:

• HTTP Request Type: Specifies the HTTP request method used for the HTTP ping.
• URL: Defines the HTTP or HTTPS target. Specify one URL up to 2000 characters long.
Consider the following when configuring HTTP performance monitors:
• Performance management must be enabled for the organization for HTTP performance monitors to function. See
Enable Performance Management (page 452) for more information about enabling performance management.
• An IKE tunnel cannot be deleted when it has an assigned performance monitor. The performance monitor must
be removed first.
• Local subnets associated with an IKE gateway connection cannot be modified when the gateway has an as-
sign performance monitor. Remote any-to-any subnets associated with an IKE gateway connection cannot be
modified when the gateway has an assign performance monitor.
• When a new probe is assigned to an IKE tunnel, traffic forwarding continues until the probe has completed the
first iteration for both tiers before declaring the tunnel up or down.

Forwarding path lists

Forwarding path lists define the uplink preferences for up to two breakout types on an NSG. These forwarding path
lists can then be used as actions in policy rules for ingress forwarding policies. Specify Forwarding Path List for the
Action on the Action tab and then select a forwarding path list from the drop-down menu. Two forwarding actions
with different uplink preferences can be configured.
Refer to the VSP User Guide for more information about creating policy rules for ingress forwarding policies.

From the enterprise view, navigate to Networks, L3 Domains -> L3 Domain -> Forwarding Path Lists.
The following parameters can be configured for forwarding path lists:
• Name
• Description
The following parameters can be configured for forwarding path list entries:
• Forwarding Action: Specifies the breakout type for the traffic flow.
• Uplink Preference: Specifies a primary or secondary uplink preference.

8.3. SaaS Routing 569

 VNS User Guide, Release 5.3.2

Note: An Uplink Preference can be specified only when Underlay PAT or Underlay Route is selected for the
Forwarding Action.

Note: If the specified breakout action cannot be enacted due to an unusable uplink or an unavaiable IKE tunnel, an
implicit drop action is applied.

8.3.6 Procedures for SaaS routing

This section provides the detailed steps required to configure VSD objects and policies required to enable the features
described in this chapter.

Configuring SaaS application types

Step 1 From the enterprise view, navigate to Applications -> SaaS Application Types.

Step 2 Select to create a new SaaS application type.

Step 3 Enter a Name and Description.
Step 4 Click Create.

Step 5 On the Network Macros tab, select to create a new network macro.
Step 6 Enter a Name, Network, and Network IPv4 / Network IPv6.
Step 7 Click Create.
Step 8 Repeats Steps 6 and 7 to create more network macros, as required.

Configuring SaaS application groups

Step 1 From the enterprise view, navigate to Applications -> SaaS Application Types -> SaaS Application
Groups.

Step 2 Select to create a new SaaS application group.

Step 3 Enter a Name and Description.
Step 4 Click Create.

Step 5 On the Associated SaaS Application Types tab, click to add SaaS application types.
Step 6 Select SaaS application types to add to the group and click Select.

Creating a Performance Monitor for HTTP Ping

Note: Before proceeding, ensure that performance management is enabled for the organization. See Enable Perfor-
mance Management (page 452) for more information about enabling performance management.

Step 1 From the enterprise view, navigate to Applications -> Performance Monitors .

8.3. SaaS Routing 570

 VNS User Guide, Release 5.3.2

Step 2 Select button to create a new performance monitor.

Step 3 Enter a Name, Description, Hold Down Timer, and Probe Type. Specify HTTP for the Probe
Type.
Step 4 Click Create. Two tiers are automatically created for the HTTP performance monitor.
Step 5 Right-click the tier 2 object and select Edit.
Step 6 Enter a Down Threshold Count, Packet Count, Probe Interval, and Timeout.

Step 7 Select the tier 1 object and select in the Destination URLs panel.
Step 8 Enter a Down Threshold Count, HTTP Request Type, URL Weight in %, URL, Packet
Count, Probe Interval, and Timeout.
Step 9 Click Create.
Step 10 Repeat Steps 7 to 9 to configure more destination URLs for tier 1, as required.

Step 11 Select the tier 2 object and select in the Destination URLs panel.
Step 12 Enter a HTTP Request Type and URL.
Step 13 Click Create.
Step 14 Repeat steps 11 to 13 to configure more destination URLs for tier 2, as required.

Assigning an HTTP performance monitor to an IKE tunnel

Step 1 From the enterprise view, navigate to Infrastructure -> Network Service Gateways -> NSG
Instance –> Network Port (Port 1 or Port 2) -> Untagged VLAN .
Step 2 Select an IKE gateway connection and click on the Performance Monitors button.

Step 3 Select to assign the performance monitor to the IKE tunnel.

Configuring forwarding path lists

Step 1 From the enterprise view, navigate to Networks, L3 Domains -> L3 Domain -> Forwarding
Path Lists.

Step 2 Select button to create a new forwarding path list.

Step 3 Enter a Name and Description.
Step 4 Click Create.

Step 5 Select button to create a new forwarding path list entry.

Step 6 Enter a Forwarding Action and Uplink Preference.
Step 7 Click Create.
Step 8 Repeat Steps 5 to 7 to create a second forwarding path list entry, if required.

8.3. SaaS Routing 571

 VNS User Guide, Release 5.3.2

8.3.7 Predefined SaaS application type updates

The VSD provides an API to import a .json file with updates to the list of network macros for the predefined SaaS
application types. The .json file does not need to include network macros that are already listed in the database, it
needs to include only the new ones to be added.
The version number and upload date of the current list of predefined network macros can be viewed under the SaaS
Applications Version section of the System Configuration tab. Navigate to Platform Configuration -> Settings ->
System Configuration .

Predefined SaaS application type update sample

The following sample shows the format for the .json file required to update the list of network macros for the predefined
SaaS applications types in the VSD.

{
"command":"SAAS_APPLICATION_TYPE",
"Parameters":{"resources":{
“version”:”1.0”,
"saasapplicationtype":[{"name":"Github",
"children": {
"enterprisenetwork": [
{
"address": "13.250.155.205",
"netmask": "255.255.255.255"
},
{
"address": "13.250.155.206",
"netmask": "255.255.255.255"
}
] }}]}}
}

8.3.8 SaaS routing CLI samples

The following show commands are supported for HTTP ping performance monitors:
• nuage-nsg-ike-cli show ipsec-stats
• ovs-appctl -t nuage-probed httping/dump
• ovsdb-client dump Nuage_IKE_Monitor_Config
• nuage-nsg-ike-cli show tunnel-status-summary
• ovs-appctl ike/list-tunnels
• ovs-appctl -t nuage_perfd httping/dump-probe-config
• ovs-appctl -t nuage_perfd httping/dump-probe-stats
• ovs-appctl -t nuage_perfd httping/dump-tunnels
• ovs-appctl -t nuage_perfd httping/dump-tunnel-state
• probe-info detail

8.3. SaaS Routing 572

 VNS User Guide, Release 5.3.2

ovs# iptables -t mangle -nvL OUTPUT

Chain OUTPUT (policy ACCEPT 992 packets, 229K bytes)
pkts bytes target prot opt in out source destination
0 0 MARK 47 -- * * 0.0.0.0/0 0.0.0.0/0
˓→ MARK xset 0xfac0000/0xfffc0003
0 0 MARK tcp -- * * 10.15.1.254 0.0.0.0/0
˓→ tcp dpt:179 MARK xset 0xfac0000/0xfffc0003
416K 22M MARK tcp -- * * 0.0.0.0/0 0.0.0.0/0
˓→ tcp dpt:80 mark match 0x402/0x402 MARK xset 0xfac0000/0xfffc0000
416K 22M RETURN tcp -- * * 0.0.0.0/0 0.0.0.0/0
˓→ tcp dpt:80 mark match 0x402/0x402
416K 47M DSCP all -- * * 10.15.1.254 50.1.1.1
˓→ mark match 0xfac0000/0xfffc0000 DSCP set 0x00
416K 47M MARK all -- * * 10.15.1.254 50.1.1.1
˓→ mark match 0xfac0000/0xfffc0000 MARK set 0x80100
56 12792 DSCP udp -- * * 0.0.0.0/0 50.1.1.1
˓→ multiport dports 500,4500 mark match 0x0 DSCP set 0x38
56 12792 MARK udp -- * * 0.0.0.0/0 50.1.1.1
˓→ multiport dports 500,4500 mark match 0x0 MARK set 0x80000
0 0 MARK udp -- * * 0.0.0.0/0 0.0.0.0/0
˓→ udp spt:4500 MARK or 0x1
0 0 CLASSIFY udp -- * * 0.0.0.0/0 0.0.0.0/0
˓→ multiport dports 3784,4784 CLASSIFY set 0:64
0 0 MARK udp -- * * 0.0.0.0/0 0.0.0.0/0
˓→ udp spt:4500 MARK or 0x1
1630K 403M DNSFORWARDER all -- * * 0.0.0.0/0 0.0.0.0/0
3231 323K CONNMARK tcp -- * * 0.0.0.0/0 0.0.0.0/0
˓→ tcp spt:893 CONNMARK restore

--------------------------------------------------------------------------------------
˓→--------------------------

ovs# nuage-nsg-ike-cli show ipsec-stats

--------------------------------------------------------------------------------------
˓→----------------------------------------------------------------

Gateway Name Local IP Remote IP encrypted_

˓→packets decrypted_packets encrypted_bytes decrypted_bytes
--------------------------------------------------------------------------------------
˓→----------------------------------------------------------------

My_Ikev2_Gw_Conn_3 10.15.1.254 50.1.1.1 488629

˓→ 328201 25859130 45642815
--------------------------------------------------------------------------------------
˓→----------------------------------------------------------------

ovs# ovs-appctl -t nuage-probed httping/dump

UUID Down cnt Success cnt URL
51e8f4a4-5fda-4abd-a3f6-fe7e6bbe4824 0 794 http://google.com

--------------------------------------------------------------------------------------
˓→--------------------------

ovs# ovsdb-client dump Nuage_IKE_Monitor_Config

Nuage_IKE_Monitor_Config table
_uuid destination_target_list down_threshold_count
˓→probe_uuid rate timeout
------------------------------------ ----------------------- -------------------- ----
˓→---------------------------------- ------- -------

7be2294d-c90b-4e1f-ab91-5e8812723c11 "http://google.com" 5
˓→"51e8f4a4-5fda-4abd-a3f6-fe7e6bbe4824" "1.000" 5000

8.3. SaaS Routing 573

 VNS User Guide, Release 5.3.2

--------------------------------------------------------------------------------------
˓→--------------------------

ovs# nuage-nsg-ike-cli show tunnel-status-summary

--------------------------------------------------------------------------------------
˓→------------------------

Gateway Name Local IP Remote IP Phase1

˓→ Phase2
--------------------------------------------------------------------------------------
˓→------------------------

My_Ikev2_Gw_Conn_3 10.15.1.254 50.1.1.1 up

˓→ up

vsc# tools vswitch 66.201.62.24 command "ovs-appctl ike/list-tunnels"

IKE tunels:
uuid skb_mark priority qos_class
˓→tunnel_status probe_status local_subnet remote_subnet
--------------------------------------------------------------------------------------
˓→---------------------------------------------------------

79e8ad4e-4203-4ce4-a85b-a6c75842795d 0xa 10 112 UP

˓→ UP 0.0.0.0/0 0.0.0.0/0

be204751-7785-47ae-be99-650c38911ed2 0x14 20 112 UP

˓→ UP 0.0.0.0/0 0.0.0.0/0

# ovs-appctl -t nuage_perfd httping/dump-probe-config

[
{
"hold_down_time": 1000,
"probe_name": "probe_ovs-2",
"probe_uuid": "e0466d0d-8f89-4328-9487-50a7d69e7f0a",
"tier_1_urls": [
{
"packet_count": 1,
"packet_interval": 1000,
"packet_timeout": 3000,
"url_request_type": "GET",
"url_string": "http://www.google.com",
"url_threshold": 5,
"url_tier": 1,
"url_uuid": "797e3854-52f7-47a2-a37a-345c29cfc0d4",
"url_weight": 100}],
"tier_2_pkt_count": 1,
"tier_2_pkt_interval": 1000,
"tier_2_request_timeout": 3000,
"tier_2_threshold": 2,
"tier_2_urls": [
{
"url_request_type": "GET",
"url_string": "http://www.google.co.in",
"url_tier": 2,
"url_uuid": "671c96e7-f544-4912-b797-67d59fe5aeea"},
{
"url_request_type": "GET",
"url_string": "http://www.yahoo.com",
"url_tier": 2,
"url_uuid": "fe39fd9a-a1d7-42f7-b474-54b85d612c4f"}],
"tunnels:" [

8.3. SaaS Routing 574

 VNS User Guide, Release 5.3.2

{
"tunnel_name": "My_Ikev2_Gw_Conn_1-2"}]}]

# ovs-appctl -t nuage_perfd httping/dump-probe-stats

[
{
"probe_name": "probe_ovs-2",
"probe_state": "UP",
"probe_uuid": "e0466d0d-8f89-4328-9487-50a7d69e7f0a",
"tier_1_stats": [
{
"failure_sum": 0,
"tier_1_url_stats": [
{
"failure_count": 1,
"failure_reason": "Probe request timed out"
"success_count": 0,
"url_string": "http://www.google.com",
"url_uuid": "797e3854-52f7-47a2-a37a-345c29cfc0d4"}],
"tier_state": "UP"}],
"tier_2_stats": {
"failure_count": 0,
"success_count": 0,
"tier_2_failures": [
{
"failure_reason": "Server returned '404 Not Found'",
"url_string": "http://www.yahoo.com",
"url_uuid": "7fe39fd9a-a1d7-42f7-b474-54b85d612c4f"}],
"tier_state": "UP"},
"tunnel_name": "My_Ikev2_Gw_Conn_1-2",
"tunnel_uuid": "0e830c02-6807-4cb7-a691-ebe9e4b800f2"}]

# ovs-appctl -t nuage_perfd httping/dump-tunnels

[
{
"probe_name": "probe_ovs-2",
"probe_uuid": "e0466d0d-8f89-4328-9487-50a7d69e7f0a",
"skb_mark": 10,
"tunnel_name": "My_Ikev2_Gw_Conn_1-2",
"tunnel_uuid": "0e830c02-6807-4cb7-a691-ebe9e4b800f2"},
{
"probe_uuid": "00000000-0000-0000-0000-000000000000",
"skb_mark": 20,
"tunnel_name": "My_Ikev2_Gw_Conn_2-2",
"tunnel_uuid": "0ffee5ef-a66c-4483-8df8-ee38525001f1"}]

# ovs-appctl -t nuage_perfd httping/dump-tunnel-state

[
{
"tunnel_name": "My_Ikev2_Gw_Conn_1-2",
"tunnel_state": "UP",
"tunnel_uuid": "0e830c02-6807-4cb7-a691-ebe9e4b800f2"},
{
"tunnel_name": "My_Ikev2_Gw_Conn_2-2",
"tunnel_state": "false",
"tunnel_uuid": "0ffee5ef-a66c-4483-8df8-ee38525001f1"}]

8.3. SaaS Routing 575

 VNS User Guide, Release 5.3.2

#### IKE TUNNEL STATE AT OVS

# nuage-nsg-ike-cli show tunnel-status-summary
-------------------------------------------------------------------------------------
˓→-------------------------

Gateway Name Local IP Remote IP Phase1

˓→ Phase2
--------------------------------------------------------------------------------------
˓→------------------------

My_Ikev2_Gw_Conn_1-2 10.15.3.254 50.1.1.1 up

˓→ up
My_Ikev2_Gw_Conn_2-2 10.15.3.254 51.1.1.1 up
˓→ up
------------------------------------------------------------------------------------
˓→--------------------------

[root@nsg-99-211-31-184 ~]#

vsc# show vswitch-controller gateway ike 192.182.137.169 probe-info detail

===============================================================================
Virtual Switch Controller Gateway IKE Information.83.137.169 probe-info detail
===============================================================================
NSG ID : 192.182.137.169
Connection Id : 79e8ad4e-4203-4ce4-a85b-a6c75842795d
IP Addr : 195.165.148.132

Probe UUID : 1f498198-811b-4cce-9fb6-79334ece99f3

Probe Name : IKEHTTPerform4
Probe Payload : 137 Hold Down Timer : 15000
DSCP Value : 0
Tier 1 Information :
Packet Count : 1 Probe Interval : 1
Timeout : 1500 Number of URLs : 2

HTTP Method : head

Percentage Weight : 100 Down Threshold Count : 5
URL : http://www.yahoo.com

HTTP Method : get

Percentage Weight : 100 Down Threshold Count : 4
URL : http://gateway.zscalerbeta.net/vpntest

Tier 2 Information :
Packet Count : 1 Probe Interval : 3
Timeout : 2000 Down Threshold Count : 5
Number of URLs : 10

HTTP Method : head

URL : https://www.tmall.com

HTTP Method : head

URL : https://www.google.co.jp

HTTP Method : get

URL : https://www.amazon.com

HTTP Method : head

URL : https://www.uol.com.br

8.3. SaaS Routing 576

 VNS User Guide, Release 5.3.2

HTTP Method : get

URL : https://www.wikipedia.org

HTTP Method : head

URL : https://www.google.com.br

HTTP Method : get

URL : https://www.google.ru

HTTP Method : head

URL : https://www.reddit.com

HTTP Method : get

URL : https://www.google.com

HTTP Method : get

URL : https://www.sohu.com
-------------------------------------------------------------------------------
NSG ID : 191.183.137.169
Connection Id : be204751-7785-47ae-be99-650c38911ed2
IP Addr : 104.129.194.39

Probe UUID : 1f498198-811b-4cce-9fb6-79334ece99f3

Probe Name : IKEHTTPerform4
Probe Payload : 137 Hold Down Timer : 15000
DSCP Value : 0
Tier 1 Information :
Packet Count : 1 Probe Interval : 1
Timeout : 1500 Number of URLs : 2

HTTP Method : head

Percentage Weight : 100 Down Threshold Count : 5
URL : http://www.yahoo.com

HTTP Method : get

Percentage Weight : 100 Down Threshold Count : 4
URL : http://gateway.zscalerbeta.net/vpntest

Tier 2 Information :
Packet Count : 1 Probe Interval : 3
Timeout : 2000 Down Threshold Count : 5
Number of URLs : 10

HTTP Method : head

URL : https://www.tmall.com

HTTP Method : head

URL : https://www.google.co.jp

HTTP Method : get

URL : https://www.amazon.com

HTTP Method : head

URL : https://www.uol.com.br

HTTP Method : get

URL : https://www.wikipedia.org

8.3. SaaS Routing 577

 VNS User Guide, Release 5.3.2

HTTP Method : head

URL : https://www.google.com.br

HTTP Method : get

URL : https://www.google.ru

HTTP Method : head

URL : https://www.reddit.com

HTTP Method : get

URL : https://www.google.com

HTTP Method : get

URL : https://www.sohu.com
-------------------------------------------------------------------------------
No. of IKE Connections: 2
===============================================================================

8.3. SaaS Routing 578

 CHAPTER

NINE

BRANCH SERVICES

• Virtual Network Function on NSG (page 580)

• WiFi on NSG (page 616)

579
 VNS User Guide, Release 5.3.2

9.1 Virtual Network Function on NSG

• Overview (page 581)

– Supported VNF features (page 582)

* Firewall (page 582)

* WAN-Optimization (page 583)
– VNF onboarding and bootstrapping (page 584)

* VNF onboarding (page 584)

* VNF Deployment (page 584)
– Datapath integration (page 584)

* Firewall datapaths (page 584)

* WAN-Optimization datapaths (page 585)
– EMS management of VNFs on NSG (page 585)
– Feature support considerations (page 585)

* General notes (page 585)

* Interface considerations (page 585)
• VNF life-cycle states and actions (page 586)
– States (page 587)
– Actions (page 588)
• Configuration specifics (page 588)
– Enabling VNF management (page 588)
– Storing and configuring input files (page 589)

* qcow2 file (page 589)

* Bootstrap ISO (page 589)
* VNF metadata XML blob (page 589)
– VSD objects and policies (page 589)

* VNF Metadata (page 589)

* VNF Threshold Policy (optional) (page 590)
* VNF Descriptor (page 590)
* VNF Interface Descriptor (page 590)
* VNF Instance (page 590)
* Redirection Targets (page 591)
* Ingress and Egress Forwarding Policies (page 591)
• Workflows for VNF on NSG features (page 591)
– Firewall workflow (page 591)

9.1. Virtual Network Function on NSG 580

 VNS User Guide, Release 5.3.2

– WAN-Optimization workflow (page 591)

• Procedure for VNF Onboarding (page 592)
– Prerequisites (page 592)
– Creating the Bootstrap ISO File for the VNF (page 592)
• VSD procedures for VNF on NSG (page 593)
– Creating the VNF Metadata (page 593)
– Creating the VNF Descriptor (page 594)
– Creating the VNF Interface Descriptors (page 596)
– Creating a VNF Instance (page 597)
– Redirecting Traffic to the VNF (firewall) (page 603)
– Redirecting Traffic to the VNF (WAN-Optimization) (page 606)
• Procedures for WAN-Optimization with Ipanema (page 608)
– Verifying the NSG partition configuration (page 608)
– Disable DWS feature (page 609)
– Define default ingress and egress ACL rules on the VNF (page 609)
– Viewing the required IP addresses in VSD (page 609)
– Ipanema system provisioning (page 610)
– Ipanema application group provisioning (page 612)
– Enable compression (page 612)
– Verify compression (page 612)
• Troubleshooting (page 612)
– CPU cycle consumption due to ARP broadcast packets (page 612)
– Error States (page 613)
• VNF metadata XML sample blobs (page 613)
– Firewall XML blob (page 613)
– WAN-Optimization XML blob (page 614)
• CLIs (page 615)

9.1.1 Overview

The NSG supports the hosting of third-party VNFs to provide branch-in-a-box functionality through the integration
of network appliances. The VSD manages the onboarding, deployment, datapath integration, and life-cycle aspects of
VNFs hosted on the NSG. Setup, configuration, and deletion can all be handled using the VSD.
Key functionality includes the following:
• Automated integration, management, and provisioning of the VNF using the VSD as the single point of inte-
gration. Although the VSD does not host the VNF image, it does manage the complexity of image storage,
evaluation of resources on the CPE, and VNF deployment.

9.1. Virtual Network Function on NSG 581

 VNS User Guide, Release 5.3.2

• Full management of VNF life-cycle from bootstrapping to service-chaining and VNF deletion.
• Intelligent service chaining at L4/L7 which enables granular management of traffic.
• Simplified packet processing through integration of the VNF datapath with VNS data plane at the host OS layer.
• Data network isolation and VNF interface management for full control of datapath configuration and security.
The solution is designed to accommodate any third-party VNFs. Day0 configuration of the VNF is performed using the
bootstrap process. Day2 configuration updates are enabled through the EMS on each VNF vendor. The management
interfaces of the VNFs and the EMS can be assigned to a separate domain. This will ensure secure VNF management
with complete isolation from the control and management of VNS.

Note: VNF features on NSG are applicable to L3 traffic only.

Supported VNF features

The VSD currently provides support for the following VNF features on the NSG:
• Firewall (page 582)
• WAN-Optimization (page 583)

Firewall

The NSG supports the hosting of firewall appliances as managed VNFs. The feature is implemented using a single
VNF for securing access networks from the network traffic in the following contexts:
• VNF insertion for traffic to underlay (local breakout to internet)
• VNF insertion for traffic to overlay

Fig. 9.1: Single Firewall VNF Use Case

9.1. Virtual Network Function on NSG 582

 VNS User Guide, Release 5.3.2

See Firewall workflow (page 591) for high-level steps to enable and manage the feature.

WAN-Optimization

The NSG supports hosting of WAN-Optimization appliances as managed VNFs. In this use case, two NSGs hosting
VNFs of type WAN-OPT are required in a bookended deployment.
Inter-VNF communication
• VIPE uses the IP address space of the VNF-VPort of VNF infra-domain.
• Inter-VIPE communication uses TCP for the computation of measurement.
• Addresses the issue of multi-tenancy for different access domains by using the parent VPort to send the probes
using UDP and the default path alone to learn the health of the WAN networks.

Fig. 9.2: Inter-VNF Communication

Dynamic WAN selection

Fig. 9.3: Dynamic WAN Selection

Dynamic WAN Steering

Dynamic WAN steering is an automatic VSD function that configures the gateway IP addresses on the EMS of the
Ipanema. The VSD displays the dummy MAC/IP Address mapped to WAN link. The VNF performs ARPs for these
IP addresses and learns the MACs from OVS.

9.1. Virtual Network Function on NSG 583

 VNS User Guide, Release 5.3.2

See WAN-Optimization workflow (page 591) for high-level steps to enable and manage the feature.

VNF onboarding and bootstrapping

VNF bootstrapping using the VSD consists of the following:

VNF onboarding

This process is typically generic between VNF features and requires management of the following data files and VSD
objects:
• qcow2 file: a file required to deploy the VNF as a KWM virtual machine
• VNF bootstrap ISO: the bootstrap configuration file for the VNF
• VNF metadata XML blob: XML file that specifies the VNF VM definition and Nuage metadata
• VNF Descriptor: a VSD object that specifies VNF VM CPU, memory, disk space, and links to the VNF metadata
XML blob and threshold policy

VNF Deployment

This process is the creation of VSD objects to specify the L3 domain/mapping and the VNF instance configuration:
• VNF Domain Mapping: specifies the segmentation type (typically VLAN) and ID for the virtual wire pairs, in
addition to automatically creating a VNF Zone and subnet for the NSG and sub ports
• VNF Instance: the logical object for the VNF in the VSD - parent to the LAN and WAN interfaces and used to
manage VNF deployment status

Datapath integration

This process varies depending on the VNF feature use-case, but typically involves configuring the a subset of the
following VSD policies:
• Redirection Targets: specify the target mapping used for VNF/LAN WAN interfaces in the Advanced Forward-
ing Policies and Ingress/Egress Forwarding Policies
• Ingress/Egress Forwarding Policies and Advanced Forwarding Rules: specify the origin/destination of the flow
and the target VNF interface
• Underlay Internet Policy Group: a special policy group to redirect traffic to the underlay (breakout to Internet)

Firewall datapaths

1 Configure redirection targets of Service Insertion Type “NSG-VNF” that map to all the required LAN
and WAN vports on each VNF. Two per VNF (one for LAN, one for WAN).
2 Configure an ingress forwarding policy and egress forwarding policy for the VNF.
3 Configure advanced forwarding rules for each policy. The number of rules depends on the number of
WANs, required security depth, and the specific use-case.
Each rule specifies:
• The origin location (subnet, policy group, or Underlay Internet Policy Group)

9.1. Virtual Network Function on NSG 584

 VNS User Guide, Release 5.3.2

• The destination network (subnet, policy group, or Underlay Internet Policy Group)
• The redirection target mapping for the required VNF interface

WAN-Optimization datapaths

1 Configure redirection targets of Service Insertion Type “NSG-VNF” that map to all the required LAN
and WAN vports on each VNF. Two per VNF (one for LAN, one for WAN).
2 Configure an ingress forwarding policy and egress forwarding policy for each VNF. Two policies per
VNF, for 4 total per 2 bookended VNFs.
3 Configure advanced forwarding rules for each policy. The number of rules depends on the number of
WANs, required security depth, and the specific use-case.
Each rule specifies:
• The origin location (of type subnet), eg subnet-20.
• The destination network (of type subnet), eg, subnet-21.
• The redirection target mapping for the required VNF interface, eg, subnet-21-wan

EMS management of VNFs on NSG

VNFs hosted on the NSG can be managed by a separate EMS. See the “VNF Solution Brief” for more information.

Feature support considerations

This section describes special information and notes about VNF on NSG support.

General notes

• VNF can be deployed on NSG-E300, NSG-X200 and NSG-X300 NSG-X.

• A single VNF per NSG is supported.
• The qualified versions of the VNFs are as follows:
– Fortinet - FortiGate versions 5.4.4 and 5.6.3
– PaloAlto – PA-VM-KVM-8.0.0
• With a Fortinet trial license, the VNF must have 1024 MB memory, 1 CPU, and 1 GB storage; however, with a
full license, different values can be provisioned.
• The VNF should be running in virtual wire mode.

Interface considerations

Each VNF instance must have exactly 3 interfaces: a management interface, a LAN interface, and a WAN interface.
Consider the following:
• The management interface cannot be in the same domain as the non-management interfaces.
• The subnet with which the management interface is associated can have no other VPorts in it.
• Both non-management interfaces must be in the same subnet.

9.1. Virtual Network Function on NSG 585

 VNS User Guide, Release 5.3.2

• PAT to underlay must be enabled on the subnet associated with the non-management interfaces.
• On the VNF data ports (non-management), it is mandatory to enable Allow address spoofing.
• It is not possible to manually create and attach a VM VPort to the VNF interface and subnet, this is done
automatically.
To redirect traffic to the VNF, in addition to creating it, advanced forwarding rules must be added.
When a VNF is created, VM VPorts are created in the VNF infrastructure domain. The name of each of these VPorts
is a concatenation of the VNF name (e.g., “test”) plus the VNF interface name (e.g., “in” or “out”, so these examples
would result in “test-in” or “test-out”). The VPort names are limited to a total of 16 characters, so that if the VNF
name is 16 characters or more, there will be no character availability left for the VNF interface name, and all the names
will look the same for VSD. The An error about conflicting VPorts will be thrown: “Another vPort exists with same
name”. To avoid this, make the VNF name sufficiently short that the VNF interface name can act as a differentiator.

Fig. 9.4: VNF VM VPort Name Construction With 16-Character Limit

9.1.2 VNF life-cycle states and actions

The life-cycle of the VNF consists of several states and actions that can be performed depending on the current state.
The following flow chart displays the relationship between the various states and actions that can be performed.

9.1. Virtual Network Function on NSG 586

 VNS User Guide, Release 5.3.2

Fig. 9.5: VNF life-cycle state and action flow

States

The VNF goes through the following states during its life-cycle. The current state can be viewed in VNF Status.
INIT
When a VNF instance is first created by selecting the VNF Descriptor and NSG to deploy it on, it is in the INIT state.
DEPLOYING
Invoking the DEPLOY action sets the VNF status to DEPLOYING. The VNF VM is being deployed to the selected
NSG.
SHUTOFF
The VNF VM is completely deployed and the START, RESTART, and UNDEPLOY actions are now available.
RUNNING
The VNF VM has successfully powered on. The STOP, UNDEPLOY, RESTART, and REDEPLOY actions can now
be invoked.

9.1. Virtual Network Function on NSG 587

 VNS User Guide, Release 5.3.2

Actions

After the INIT state, the following actions can be performed by selecting the VNF instance, choosing an action, and
clicking Invoke:
DEPLOY
This action involves downloading the QCOW and ISO image locally from a remote server. Deployment creates the
VM’s XML configuration from the metadata and descriptors. The VM definition is validated to ensure readiness for
the START action. If any validation fails, then an error is presented. The VM is defined and placed in SHUTOFF state
(which indicates that the VM is deployed but not started). The user can then invoke the START action on the VM
VNF.
REDEPLOY
This action is required when the QCOW or ISO fields are changed after deployment of the instance.
START
The action starts the VM and updates its status in the VSD. If the VM starts successfully, it is transferred to the
RUNNING state.
RESTART
This action is required when the values for Memory, Storage, and/or CPU Count are changed.
STOP
This action triggers a graceful shutdown of the VM, followed by a hard shutdown after a timeout period. The VM is
placed in the SHUTOFF state.
UNDEPLOY
This action can be performed at the SHUTOFF and RUNNING stages of the VNF life-cycle. It stops a VM, undefines
it, and deletes the VPorts and disk images associated with the VM. It is equivalent to deleting the instance.

Note:
• There is also a sync button on the VNF instance that probes for status refresh
• VPorts will be unresolved when a VNF is successfully undeployed
• Once deployed, subnet association cannot be changed. If the subnet association needs to be changed, undeploy,
change the subnet association and then redeploy.
• To reboot a VNF, do a stop and start.

9.1.3 Configuration specifics

This section describes the VSD configuration tasks required to enable VNF management, provision the required net-
work objects for VNF onboarding and deployment, and configure the required policy objects and groups.

Enabling VNF management

To enable VNF support, the csproot user creates or edits the appropriate organization profile by enabling the VNF
Management checkbox on the Organization Profile popup.

9.1. Virtual Network Function on NSG 588

 VNS User Guide, Release 5.3.2

Storing and configuring input files

The following subsections describe specifics about the input files for VNF onboarding.

qcow2 file

The qcow2 file must include an MD5 checksum and be on a server accessible by the NSG, the URI of which is
specified in the metadata XML blob.

Bootstrap ISO

The bootstrap configuration for the VNF is created as an ISO and includes an MD5 checksum. These files are made
available in the same location as the qcow2 image and checksum. The bootstrap configuration has information about
virtual wire pairs that are created and a basic security policy allowing all traffic between the LAN (in) and WAN (out)
ports of the VNF.

VNF metadata XML blob

See VNF metadata XML sample blobs (page 613) for an example of the XML content. Only edit the following fields:
• image-uri
• image-md5-checksum
• iso-uri
• iso-md5-checksum

Note: If HTTPS is used in the VNF metadata blob for QCOW and bootstrap ISO, do not use an IP address; instead,
use the FQDN.

Note: Changing the metadata is permitted, even after it has been associated with an NSG.

VSD objects and policies

The following VSD objects and policies are relevant to VNF on NSG.

VNF Metadata

The VNF Metadata object is the container for the metadata XML blob and is referenced by the VNF Descriptor.

Navigate to Platform Configuration -> Applications -> VNF Metadata .

9.1. Virtual Network Function on NSG 589

 VNS User Guide, Release 5.3.2

VNF Threshold Policy (optional)

The threshold policy defines the limit for different resource utilization of the VNF VM on the NSG. The threshold
policy is defined at the csproot level and is applied to a VNF Descriptor. Consider creating different threshold policies
for different NSG variants for a VNF.
If no threshold policy is selected during the VNF Descriptor creation, the VNF Descriptor is associated with the default
VNF Threshold Policy.

Navigate to Platform Configuration -> Applications -> VNF Threshold Policy .

Note: The assigned threshold policy is applied only after the VNF is in the running state. The policy is not enforced
when the VNF is being deployed.

VNF Descriptor

The VNF Descriptor specifies the VNF type for the required feature (such as Firewall or WAN-OPT), the vendor,
memory/CPU parameters, and links to the VNF Metadata and Threshold Policy.

Navigate to Platform Configuration -> Applications -> VNF Descriptor .

VNF Interface Descriptor

A VNF Interface Descriptor is a logical object that represents an interface on the VNF. There are three types:
• MANAGEMENT: the management interface
• LAN: the LAN interface (trusted)
• WAN: the WAN interface (untrusted)
All three interface types must be configured for the VNF.

Select a VNF Descriptor and select or to add an interface descriptor.

VNF Instance

The VNF Instance is the logical object that represents the VNF. Life-cycle management actions can be performed by
selecting the VNF Instance.
As soon as the first VNF is instantiated, a VNF template appears in the template listings for that organizations. All other
VNFs are instantiated from this template, which behaves like other templates: it cannot be edited or deleted until all
its instantiations have been destroyed. Whenever a VNF is instantiated, the zone and the VPorts, etc. corresponding
to the metadata and interfaces set up by csproot are automatically created, appearing in the Networks view for the
organization admin.

Navigate to Applications -> VNF Instance and click on .

Note: Creating the first VNF instance creates a VNF infrastructure domain. Subsequent VNF instance creation does
not create additional infrastructure domains.

9.1. Virtual Network Function on NSG 590

 VNS User Guide, Release 5.3.2

Redirection Targets

Redirection Targets are logical objects that represent vports and are pointed to by Advanced Forwarding Rules as the
destination for redirected traffic.
Navigate to Platform Configuration -> Applications -> Layer 3 Domains -> Policies -> Redirection Targets.

Ingress and Egress Forwarding Policies

Enables traffic to be redirected or to be modified for traffic management purposes by matching specific criteria. Mir-
roring Ingress Forwarding Policies and Egress Forwarding Policies, along with Advanced Forwarding Rules (the
Forwarding Policy content), must be configured at the NSG level to control traffic going in and out of the VNF.
Ingress and Egress Forwarding Policies: Navigate to Platform Configuration -> Applications -> Layer 3 Domains ->
Policies -> <Policy Type>

Advanced Forwarding Rules: Select an Ingress or Egress Forwarding Policy and select or .

9.1.4 Workflows for VNF on NSG features

This section provides high-level workflows that list the procedures required to configure and enable VNF on NSG
features. Where applicable, the workflows link to the specific procedure for the object or task.

Firewall workflow

The following steps describe the process to enable the firewall feature.
1 Create the bootstrap ISU file by performing Creating the Bootstrap ISO File for the VNF (page 592).
2 Create the VNF metadata blob by performing Creating the VNF Metadata (page 593).
3 Configure the VNF Descriptor by performing Creating the VNF Descriptor (page 594).
4 Configure the VNF Interface Descriptors for Mgmt, LAN, and WAN by performing Creating the VNF
Interface Descriptors (page 596).
5 Configure the VNF Instance by performing Creating a VNF Instance (page 597).
6 Configure the datapaths required for firewall by performing Redirecting Traffic to the VNF (firewall)
(page 603).

WAN-Optimization workflow

The following steps describe the process to enable the WAN-Optimization feature.
1 Create the bootstrap ISO file by performing Creating the Bootstrap ISO File for the VNF (page 592).
2 Create the VNF metadata blob by performing Creating the VNF Metadata (page 593).
3 Configure the VNF Descriptor by performing Creating the VNF Descriptor (page 594).
4 Configure the VNF Interface Descriptors for Mgmt, LAN, and WAN by performing Creating the VNF
Interface Descriptors (page 596).
5 Configure the VNF Instance by performing Creating a VNF Instance (page 597).

9.1. Virtual Network Function on NSG 591

 VNS User Guide, Release 5.3.2

6 Configure the datapaths required for WAN-Optimization by performing Redirecting Traffic to the VNF
(WAN-Optimization) (page 606).
Continue steps specific to WAN-Optimization on Ipanema:
7 Perform all procedures in Procedures for WAN-Optimization with Ipanema (page 608).

9.1.5 Procedure for VNF Onboarding

This procedure is performed by csproot.

Prerequisites

• QCOW image uploaded to a server along with the associated MD5 checksum.
• Bootstrap configuration for the VNF created as an ISO file (instructions below) and uploaded to the same server
along with the associated MD5 checksum.

Creating the Bootstrap ISO File for the VNF

The sample configuration included below makes a virtual wire pair on port2 and port3 and allows all the traffic across
them. To configure different features, get the appropriate licenses from Fortinet.
Fortinet requires the bootconfig to be provided as a disk drive in the cloud init format. This is the reason the ISO file
is created with the specified configuration structure.
To create the ISO file the example uses xorriso, but any application can be used. Whichever application is used must
be installed before beginning this procedure.
To create a bootstrap configuration for a Fortinet VNF
Step 1 Create a folder (using any name), for example bootconfig. Under that folder, create a folder
hierarchy and the files as displayed in the output below, using exactly the same names:

[root@util-1 latest]# pwd

/root/bootconfig/openstack/latest
[root@util-1 latest]# ls
meta_data.json network_data.json user_data vendor_data.json
[root@util-1 latest]# cat meta_data.json
{}
[root@util-1 latest]# cat network_data.json
{}
[root@util-1 latest]# cat vendor_data.json
{}
[root@util-1 latest]# cat user_data
config system interface
edit "port1"
set mode dhcp
next
end
config router static
edit 1
set device "port1"
next
end
config system dns
set primary 8.8.8.8

9.1. Virtual Network Function on NSG 592

 VNS User Guide, Release 5.3.2

set secondary 9.9.9.9

end
config system virtual-wire-pair
edit "test"
set member "port2" "port3"
set wildcard-vlan enable
next
end
config firewall policy
edit 1
set srcintf "port2"
set dstintf "port3"
set srcaddr "all"
set dstaddr "all"
set action accept
set schedule "always"
set service "ALL"
set utm-status enable
next
edit 2
set srcintf "port3"
set dstintf "port2"
set srcaddr "all"
set dstaddr "all"
set action accept
set schedule "always"
set service "ALL"
next
end
[root@util-1 latest]#

Step 2 Create an ISO file using any name, for example boot.iso as follows (our example uses xorriso
but it can be any other application):

xorrisofs -r -J -o <boot.iso> bootconfig/

Step 3 For basic configuration, refer to the Fortigate VM Initial Configuration section in the Fortinet
documentation at: http://docs.fortinet.com/uploaded/files/1734/fortigate-vm-install-50.pdf

9.1.6 VSD procedures for VNF on NSG

This section provides the detailed steps required to configure VSD objects and policies required to enable the features
described in this chapter.

Creating the VNF Metadata

Step 1 As CSPRoot, navigate to Platform Configuration -> Applications -> VNF Metadata . In the
panel, click on . The VNF Metadata popup appears.
Step 2 In the VNF Metadata popup, enter Name, Description and libvirt XML Blob. See to VNF meta-
data XML sample blobs (page 613) for examples based on feature type. The primary attributes in
the blob are:
• Location of the VM image and the MD5 checksum.

9.1. Virtual Network Function on NSG 593

 VNS User Guide, Release 5.3.2

• Location of the sample ISO and the MD5 checksum.

• Disk parameters

Fig. 9.6: VNF Metadata

VSD validates the required attributes and populates some of the variables configured via the VNF
Descriptor (e.g. memory, storage, CPU count).

Creating the VNF Descriptor

Step 1 As CSPRoot, navigate to Platform Configuration -> Applications -> VNF Descriptor . In the
panel, click on .
Step 2 Enter Name, Description, Vendor, Memory (MB) 1024, Storage (GB) 1, CPU Count 1. Enter
the Meta URI by clicking the link to select the VNF Metadata created in the previous procedure.
The NEW VNF Descriptor popup appears. Select “Firewall” or “WAN-OPT” as required by the
type of feature deployment.

9.1. Virtual Network Function on NSG 594

 VNS User Guide, Release 5.3.2

9.1. Virtual Network Function on NSG 595

Fig. 9.7: VNF Descriptor
 VNS User Guide, Release 5.3.2

Creating the VNF Interface Descriptors

The following interfaces are required:

• a management interface
• an interface for the LAN side/Trusted
• an interface for the WAN side/Untrusted

Step 1 Select a VNF Descriptor and select or to add an interface descriptor.

Step 2 Enter a Name and select the interface type from the dropdown list of that name.

Fig. 9.8: VNF Interface Descriptor

Step 3 Create the other two interfaces the same way. After the three interfaces are created, a list similar
to the following is displayed:

9.1. Virtual Network Function on NSG 596

 VNS User Guide, Release 5.3.2

Fig. 9.9: VNF Interfaces

Creating a VNF Instance

Step 1 As organization admin, navigate to Applications -> VNF Instance and click on .
Step 2 In the New VNF popup, enter Name and Description.
Step 3 Click the paperclip icon to select both of the following:
• VNF Descriptor and
• Network Service Gateway on which the VNF is to be deployed. This must be an NSG-X and
must have the personality type “NSG”.
as shown in the screenshot below:

9.1. Virtual Network Function on NSG 597

 VNS User Guide, Release 5.3.2

Fig. 9.10: New VNF Instance

Step 4 For the management interface created, select icon to attach a domain and a subnet, each
selected from lists of Domains/Subnets for that organization. Manually assign the domain and subnet
to the management interface by double-clicking on the instance’s VNF management interface - VNF
Interface - and give the names of the domain and subnet.

9.1. Virtual Network Function on NSG 598

 VNS User Guide, Release 5.3.2

Fig. 9.11: Attach Subnet to VNF Instance Interface

For the other two interfaces, the domain and subnet mapping is created automatically.
The following screenshot captioned “Auto-created VNF Infrastructure Domain” shows the domain
that is auto-created when the first VNF instance is created. This domain contains auto-created VNF
Zone, Subnet and parent ports for both untrusted and trusted VNF interfaces.

Fig. 9.12: Auto-created VNF Infrastructure Domain

Step 5 The auto-created parent ports for the VNF data interfaces are not used for service chaining of user-
traffic. The VNF sub-VPorts that are created in the following two steps are used for user-traffic. In
the Networks > Design view, Layer 3 Domains, select the domain the traffic is to traverse and in
the panel on the right, select VNF Domain Mapping and click as shown in the screenshot below

9.1. Virtual Network Function on NSG 599

 VNS User Guide, Release 5.3.2

captioned “VNF Domain Mapping”.

Fig. 9.13: VNF Domain Mapping

Step 6 In the New VNF Domain Mapping popup, for the Segmentation Type, select VLAN. The Seg-
mentation ID is the ID of the VLAN where the virtual wire pair is to be located. This ID is unique
per domain per NSG. Select the Network Services Gateway on which this is to be configured.
When you click Create, a new VNF zone is created within the L3 domain. It will have the cor-
responding auto-created subnet and the sub-ports of that VNF. The following screenshot captioned
“VNF Domain Mapping Plus Auto-created VNF Zone” shows a VNF Domain Mapping object con-
taining the segmentation ID and the associated NSG. This also shows that once a Domain VNF
Mapping object is created, a VNF Zone containing a subnet and the corresponding VNF sub-VPorts
is auto-created.

9.1. Virtual Network Function on NSG 600

 VNS User Guide, Release 5.3.2

Fig. 9.14: VNF Domain Mapping Plus Auto-created VNF Zone

Optionally, view the XML Blob that was created (with all the attributes populated) by clicking the icon on the
VNF instance.

Fig. 9.15: Blob at VNF Instance Level

Work with the new VNF instance in the Networks > Design view. Each new instance has an automatically created zone
with subnets and VPorts. The attributes of any of these objects can be examined by selecting the object and viewing
the information in the panel on the far right, as shown in the screenshot below captioned “VNF Interface Details.”

9.1. Virtual Network Function on NSG 601

 VNS User Guide, Release 5.3.2

Fig. 9.16: VNF Interface Details

Once the VNF is running, the VNF instance object turns green and its status displays as running, as shown by the
following:

Fig. 9.17: VNF Instance Up And Running

9.1. Virtual Network Function on NSG 602

 VNS User Guide, Release 5.3.2

Redirecting Traffic to the VNF (firewall)

Once the VNF is defined, traffic from multiple subnets of the NSG on which the VNF is hosted can be steered
through the VNF. For example, the origin location can be a PG containing VPorts from multiple subnets. You can
have multiple ACLs for each of which the origin location is a different subnet. The destination can be Underlay
Internet Policy Group or it can be an overlay destination (subnet/PG, etc.). These examples are shown in the
screenshots below. Options other than subnet and policy groups for source and subnet, policy groups and Underlay
internet policy group have not been validated for routing traffic through the VNF

Fig. 9.18: VNF Traffic Flow Example for Firewall 1

9.1. Virtual Network Function on NSG 603

 VNS User Guide, Release 5.3.2

Fig. 9.19: VNF Traffic Flow Example for Firewall 2

Configure advanced forwarding rules using redirection targets as follows (refer to service chaining chapter for detailed
workflow):
Step 1 Define Redirection Targets in the VNF domain. Redirection Target Type must be NSG-VNF.
Navigate to Platform Configuration -> Applications -> Layer 3 Domains -> Policies -> Redirection
Targets.

9.1. Virtual Network Function on NSG 604

 VNS User Guide, Release 5.3.2

Fig. 9.20: Redirection Target on VNF

Two redirection targets are required, one with the untrusted VPort (WAN) and the other with the
trusted VPort (LAN).
Step 2 Define the Ingress and Egress Forwarding Policies. Navigate to Platform Configuration -> Appli-
cations -> Layer 3 Domains -> Policies -> <Policy Type>. In the New Ingress Forwarding Policy
screen, an example of which is shown below in the screenshot captioned “Ingress Forwarding Pol-
icy,” multiple subnets can be selected, but the source subnet has to be where the VNF is instantiated,
while the destination is the underlay internet policy group.

Fig. 9.21: Ingress Forwarding Policy

For example, select subnet 1 as the origin location and you can also select subnet 2, because it has
the VNF as the source but the destination network can be an underlay internet policy group or an
overlay subnet.
Step 3 The Egress Forwarding Policy is the inverse of the Ingress described above. The redirection target
will be the other untrusted port.

9.1. Virtual Network Function on NSG 605

 VNS User Guide, Release 5.3.2

Step 4 Define an Advanced Forwarding Rule by selecting an Ingress or Egress Forwarding Policy and
selecting or and specifying:
• Origin as the NSG subnet that has the non-management VNF interfaces
• Destination as the Underlay Internet Policy Group
• Redirect action to the trusted interface RT created in Step 1.
Step 5 Define a second Advanced Forwarding Rule and specify:
• Origin as the Underlay Internet Policy Group
• Destination as the NSG subnet which has the non-management VNF interfaces
• Redirect action to the untrusted interface RT created in Step 1.

Redirecting Traffic to the VNF (WAN-Optimization)

Once the VNF is defined, traffic from multiple subnets of the NSG on which the VNF is hosted can be steered through
the VNF. You can have multiple ACLs for each of which the origin location is a different subnet. For example, for
WAN A, which only handles UDP, incoming traffic gets directed to a specific vport on the VNF, which will direct it
through the matching egress vport. The second redirect rule then routes traffic from that vport to the correct WAN-A
in between the two NSGs. Complementary rule sets ensure that the same process occurs on the bookended VNF. See
the following examples of forwarding policy configuration:

Fig. 9.22: VNF Traffic Flow Example for WAN-Optimization 1

9.1. Virtual Network Function on NSG 606

 VNS User Guide, Release 5.3.2

Fig. 9.23: VNF Traffic Flow Example for WAN-Optimization 2

In another example, if traffic is being sent from ovs20 (subnet-1) to ovs21 (subnet21), the redirection targets would
resemble the following:
• Traffic from ovs-20-net-1 to ovs-21-net-1 redirect to ovs20-lan
• Traffic from ovs-21-net-1 to ovs-20-net-1 redirect to ovs20-wan
• Traffic from ovs-21-net-1 to ovs-20-net-1 redirect to ovs-21-lan
• Traffic from ovs-20-net-1 to ovs-21-net-1 redirect to ovs-21-wan
Step 1 Define Redirection Targets in the VNF domain. Redirection Target Type must be NSG-VNF.
Two redirection targets are required for each bookended VNF, one with the untrusted VPort (WAN)
and the other with the trusted VPort (LAN). Navigate to Platform Configuration -> Applications ->
Layer 3 Domains -> Policies -> Redirection Targets.
Step 2 Define the Ingress and Egress Forwarding Policies. Navigate to Platform Configuration -> Appli-
cations -> Layer 3 Domains -> Policies -> <Policy Type>. An Ingress and Egress policy must be
defined for each bookended VNF, for a total of four policies.
Step 3 Define Advanced Forwarding Rules by selecting an Ingress or Egress Forwarding Policy and
selecting or . The number of Advanced Forwarding Rules required depends on the use-case.
Specify the following:

9.1. Virtual Network Function on NSG 607

 VNS User Guide, Release 5.3.2

• Origin as an NSG subnet that has the non-management VNF interfaces

• Destination as an NSG subnet on the NSG that is hosting the other bookended VNF.
• Redirect action to the corresponding interface on the other VNF.

9.1.7 Procedures for WAN-Optimization with Ipanema

This section describes the specific configuration tasks required for using the WAN-Optimization feature with Ipanema.
Perform these tasks after performing the provisioning tasks in the previous section.

Verifying the NSG partition configuration

The partition configuration for the NSG is specified by the XML blob. You can perform the following commands on
the NSG to verify the configuration.
Step 1 As the root user, execute the lvdisplay command. The output should resemble the following:

[root@nsg nuage]# lvdisplay

LV Path /dev/vg_nsgdisk/lv_usrdisk
LV Name lv_usrdisk
VG Name vg_nsgdisk
LV UUID 1P6Syx-Uomf-dGeY-GLCY-d6Ok-KoQA-bm0E3e
LV Write Access read/write
LV Creation host, time localhost, 2018-07-06 15:56:19 -0700
LV Status available
# open 1
LV Size 39.00 GiB
Current LE 9984
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 8192
Block device 253:3
--- Logical volume ---
LV Path /dev/vg_nsgdisk/lv_wandisk
LV Name lv_wandisk
VG Name vg_nsgdisk
LV UUID laGAS9-4cx5-H0cc-Db9L-ok8Q-SO9W-COKNhO
LV Write Access read/write
LV Creation host, time localhost, 2018-07-06 15:56:20 -0700
LV Status available
# open 2
LV Size 79.00 GiB
Current LE 20224
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 8192
Block device 253:4

Step 2 As the nuage user, execute the mount | grep nsg command. The output should be the fol-
lowing:

-bash-4.2$ mount | grep nsg

/dev/mapper/vg_nsgdisk-lv_usrdisk on /home/root/vnf type ext4
(rw,nosuid,noexec,relatime,data=ordered)

9.1. Virtual Network Function on NSG 608

 VNS User Guide, Release 5.3.2

/dev/mapper/vg_nsgdisk-lv_usrdisk on /etc/libvirt/qemu type ext4

(rw,nosuid,noexec,relatime,data=ordered)
/dev/mapper/vg_nsgdisk-lv_wandisk on /home/root/wan type ext4
(rw,nosuid,noexec,relatime,data=ordered)

Disable DWS feature

The Ipanema DWS feature interferes with the WAN-Optimization feature and must be disabled. Perform the following
steps in the Ipanema interface:
Step 1 In Domain Configuration, navigate to System Provisioning -> Tools.
Step 2 Select the Advanced configuration tab and click Edit.
Step 3 Add the following line to the end:

obps_max_nap_vlans = 0

Step 4 Click Save.

Define default ingress and egress ACL rules on the VNF

Step 1 Log in to the Ipanema CLI for the VNF.

Step 2 Enter the following commands :

ipconfig natmgt none

ipconfig lan -a <VSD_VIP> -m 255.255.255.240 -g <GW_VIP>

ipconfig mgt -a <mgmt_IP> -m 255.255.255.0

salsaconfig -url https://92.222.79.96:8443 -dom Nuage2

route add -net 92.222.79.96 netmask 255.255.255.255 <mgmt_evpn_gw_IP>

reboot

where
VSD_VIP is the virtual IP dynamically assigned by the VSD
GW_VIP is the virtual gateway IP dynamically assigned by the VSD
mgmt_IP is the VNF management IP
mgmt_evpn_gw_IP is the management IP of the EVPN gateway

Viewing the required IP addresses in VSD

The Ipanema provisioning steps require several IP addresses. The following steps describe where they can be found
in the VSD.
Step 1 Main public IP address used in appliance configuration:
1. In the VSD, navigate to Networks -> Design view and select the required L3 domain for the
NSG.

9.1. Virtual Network Function on NSG 609

 VNS User Guide, Release 5.3.2

2. Select the management subnet for the NSG.

3. Select the VNF Interface object to the right of the subnet.
4. In the object selection view on the right, the IP address value is visible.
Step 2 Auxiliary public IP address used in appliance configuration:
1. In the VSD, navigate to Networks -> Design view and select the required L3 domain for the
NSG.
2. Click on the VM Vport that corresponds to the WAN interface for the NSG.
3. In the object selection view on the right, the IP address is visible.
Step 3 CPE IP addresses used when adding WAN access to an appliance:
1. In the VSD, navigate to Networks -> Design view and select the required L3 domain for the
NSG.
2. Select the NSG and click the Default Gateway icon in the right-hand window.
3. In the object selection view on the right, the Gateway IP values are visible.

Ipanema system provisioning

Perform the following steps in the Ipanema SALSA interface. See Viewing the required IP addresses in VSD (page 609)
for information about how to find the values for the main and auxiliary public/private IPs.
Step 1 Navigate to System provisioning -> Appliances and define two sites as appliances. Name values
are examples only.
In main:
• Site name: SITE_A
• Name: VIPE_1
• Main public and private IP address: the IP address of management VNF interface (enter the same IP for both)
• Auxiliary public and private IP address: the IP address of the VM VIP for the NSG (enter the same IP for both)
• Auto-reporting: Enabled
• Reversed: Enabled
In Services:
• Application Control: Enabled
• Compression: Enabled
Repeat this step for SITE_B / VIPE_2.
Step 2 Navigate to System provisioning -> Topology subnets and define 4 topology subnets. Associate
two subnets with each site. See the following example configuration:
Subnet 1:
• Name: 2_1_1_0
• Network prefix: 2.1.1.0
• Prefix length: 24
• Associated site: SITE_A
Subnet 2:

9.1. Virtual Network Function on NSG 610

 VNS User Guide, Release 5.3.2

• Name: 2_1_2_0
• Network prefix: 2.1.2.0
• Prefix length: 24
• Associated site: SITE_A
Subnet 3:
• Name: 2_2_1_1
• Network prefix: 2.2.1.1
• Prefix length: 24
• Associated site: SITE_B
Subnet 4:
• Name: 2_2_2_0
• Network prefix: 2.2.2.0
• Prefix length: 24
• Associated site: SITE_B
Step 3 Navigate to System provisioning -> WAN and define a second MPLS WAN. In the example
configurations shown in this procedure, “MPLS2” is used.
Step 4 Navigate to System provisioning -> WAN access and define WAN access. Name values used are
examples.
WAN access 1:
• Name: MyMPLS
• WAN: MPLS
• All ingress/egress B/W: 10,000
WAN access 2:
• Name: MyMPLS2
• WAN: MPLS2
• All ingress/egress B/W: 10,000
Step 5 Navigate to System provisioning -> Appliances and open the appliances created in step 1 for
modification. Attach the WAN access objects created in step 4 to the appliances. Specify the fol-
lowing in WAN Selection Configuration:
• WAN Selection: CPE
• Attach MyMPLS and MyMPLS2 and set the CPE to the IP addresses for the virtual gateways
(see step 3 in Viewing the required IP addresses in VSD (page 609)).
Step 6 Navigate to System provisioning -> Time synchronization and set Time server directory and
Synchronization server directory to:
VIPE_2 ->-> VIPE_1

9.1. Virtual Network Function on NSG 611

 VNS User Guide, Release 5.3.2

Ipanema application group provisioning

Application group provisioning allows you to specify the types of traffic that should be routed to a specific WAN
access point. Compression must be enabled.
Step 1 Navigate to Application provisioning -> Application groups and define an application group.
Specify the following:
In main:
• Name: specify a value
• Compress: enabled
In the Dictionary filters tab, move the traffic type to the right-hand window (for example, ICMP or TCP).
In the Dynamic WAN Selection tab:
• Return path: Default
• Decision Level: Per packet
• Last resort: Forward
• For Primary WAN, move the desired WAN access (in the example, either MPLS or MPLS2) to the right-hand
window.
Step 2 Repeat for all required traffic types.

Enable compression

Ensure that compression is enabled for all applicable object types.

Step 1 Make sure that the 2 appliances have Compression enabled (see Services at the bottom of Appli-
ance definition window).
Step 2 Click Service activation and enable the radio button for Compression.
Step 3 Ensure that “Compress” is enabled for all application groups (see Ipanema application group
provisioning (page 612)).

Verify compression

Step 1 After enabling compression, go to the Dashboard view and verify that under “Appliance”, the
Compression indicator is green.
Step 2 Under Dashboard -> Flows, verify that LAN to WAN traffic is being compressed on the WAN
side for traffic types corresponding to the configured application groups.

9.1.8 Troubleshooting

This section describes information relevant to troubleshooting VNF on NSG.

CPU cycle consumption due to ARP broadcast packets

ARP broadcast packets are forwarded on the VNF VPorts. Therefore, when traffic is sent after the VNF is brought up,
the ARP for the gateway sent by the VPort is sent to VNF VPorts, and the result is a broadcast storm and CPU cycle
consumption. These broadcast packets will also utilize bandwidth of the ports that are part of the subnet on which
VNF Vports are present. To avoid these issues, do one of the following:

9.1. Virtual Network Function on NSG 612

 VNS User Guide, Release 5.3.2

• Configure a policy on the VNF to block that traffic.

• Resolve all the access VPorts before “starting” a VNF. If a VPort needs to be resolved after the VNF has been
started, stop the VNF and restart it.

Error States

During VNF onboarding and operations, errors may be reported in various action contexts. The following table
provides a list of the potential errors that a user may see, the associated context in which the error is seen, and the
potential action a user may take to remediate the situation.

Table 9.1: VNF-related Errors

Error Context Followup Action
Metadata Validation failed Deploy VNF Check metadata
Requested CPU resources are not Deploy VNF Change CPU config
available
Requested memory resources are Deploy VNF Change memory config
not available
Requested storage resources are Deploy VNF Change storage config
not available
Image download failed Deploy VNF Check image location
Image MD5 checksum failed Deploy VNF Check image file
Boot config image download failed Deploy VNF Check ISO image file
VM already defined Deploy VNF Warning only
VNF libvirt parsing failed Deploy VNF Check metadata and retry
Machine is not supported Deploy VNF Correct machine type in metadata
NSG is not yet bootstrapped Deploy on Bootstrap the NSG
unbootstrapped NSG
VM already running Start VNF Transition to another state
VM already stopped Start VNF Transition to another state
Could not connect to libvirt Start VNF Contact Admin
Libvirt startup error Start VNF Contact Admin
VM not defined Start/Stop VNF Deploy VNF first
VNF resource lock acquisition Any action Retry action
failed
VNF agent was killed Undeloy VNF Intermediate warning. Should be followed by
undeploy success
Unknown error reported Anytime Contact Admin

9.1.9 VNF metadata XML sample blobs

This section contains VNF metadata sample XML blobs.

Firewall XML blob

<?xml version="1.0"?>
<domain type="kvm">
<metadata>
<nuage-vnf-metadata>
<image-uri>http://10.10.110.10/fortios.qcow2</image-uri>

9.1. Virtual Network Function on NSG 613

 VNS User Guide, Release 5.3.2

<image-md5-checksum>2602fd0c79dd1a69c14b0b46121c875e</image-md5-checksum>
<iso-uri>http://10.10.110.10/boot.iso</iso-uri>
<iso-md5-checksum>4491de582d42b95d46a84ea7dc4718d1</iso-md5-checksum>
</nuage-vnf-metadata>
</metadata>
<uuid>$VNF_UUID</uuid>
<memory unit="MiB">$VNF_MEMORY</memory>
<name>$VNF_NAME</name>
<vcpu placement="static">$VNF_VCPU</vcpu>
<os>
<type arch="x86_64" machine="pc-i440fx-rhel7.0.0">hvm</type>
<boot dev="hd"/>
<boot dev="cdrom"/>
<bootmenu enable="yes" timeout="3000"/>
</os>
<features>
<acpi/>
<apic/>
</features>
<devices>
<disk device="disk" type="file">
<driver cache="none" name="qemu" type="qcow2"/>
<source file="${VNF_IMAGE_LOCATION}"/>
<target dev="sda" bus="virtio"/>
</disk>
<disk device="cdrom" type="file">
<driver name="qemu" type="raw"/>
<source file="${VNF_ISO_LOCATION}"/>
<target bus="ide" dev="hdc"/>
<readonly/>
</disk>
<console type="pty" tty="/dev/pts/2">
<source path="/dev/pts/2"/>
<target type="serial" port="0"/>
<alias name="serial0"/>
</console>
</devices>
</domain>

WAN-Optimization XML blob

<?xml version="1.0" encoding="UTF-8"?>

<domain type="kvm">
<metadata>
<nuage-vnf-metadata>
<image-uri>http://10.10.110.10/ipanema.qcow2</image-uri>
<image-md5-checksum>1ec90e423eeeb1ad3c78671a1222db49</image-md5-checksum>
<iso-uri>http://10.10.110.10/ipanema.iso</iso-uri>
<iso-md5-checksum>d41d8cd98f00b204e9800998ecf8427e</iso-md5-checksum>
</nuage-vnf-metadata>
</metadata>
<uuid>$VNF_UUID</uuid>
<memory unit="MiB">$VNF_MEMORY</memory>
<os>
<type arch="x86_64" machine="pc">hvm</type>

9.1. Virtual Network Function on NSG 614

 VNS User Guide, Release 5.3.2

<boot dev="hd"/>
<boot dev="cdrom"/>
<bootmenu enable="yes" timeout="3000"/>
</os>
<features>
<acpi/>
<apic/>
<pae/>
</features>
<devices>
<emulator>/usr/libexec/qemu-kvm</emulator>
<disk type="file" device="disk">
<driver name="qemu" type="qcow2" cache="none"/>
<target dev="sda" bus="ide"/>
<source file="${VNF_IMAGE_LOCATION}"/>
</disk>
<disk type="file" device="cdrom">
<driver name="qemu" type="raw"/>
<target dev="hdc" bus="ide"/>
<readonly/>
<source file="${VNF_ISO_LOCATION}"/>
</disk>
<disk type="block" device="disk">
<driver name="qemu" type="raw" cache="none"/>
<source dev="/dev/disk/by-label/WANDISK"/>
<target dev="vda" bus="virtio"/>
<alias name="virtio-disk0"/>
</disk>
<console type="pty" tty="/dev/pts/1">
<source path="/dev/pts/1"/>
<target type="serial" port="0"/>
<alias name="serial0"/>
</console>
</devices>
<name>$VNF_UUID</name>
<title>$VNF_NAME</title>
<vcpu placement="static">$VNF_VCPU</vcpu>
</domain>

9.1.10 CLIs

• sudo virsh list

• sudo virsh console <id from list>
• sudo virsh dumpxml <vm name>

9.1. Virtual Network Function on NSG 615

 VNS User Guide, Release 5.3.2

9.2 WiFi on NSG

• Overview (page 616)

• Provisioning Workflow (page 617)
– Create WiFi Interface (page 617)
– Create Captive Portal Profile (page 619)
– Edit/Create SSID Connection (page 621)
– Attach Host/Bridge vPort (page 623)
– Define DHCP Pool (page 623)
– End User Workflow (page 624)
• AP Management (page 625)
• Statistics (page 625)
• Sample CLI Output (page 626)

9.2.1 Overview

The WiFi interface is a new type of Access Port on the NSG. A service provider or enterprise can deploy WiFi-enabled
NSGs in branch locations to provide Access Point (AP) capabilities. VNS enables integrated management of the WiFi
AP and UE policy enforcement from the VSD.
In the current release, the supported AP is an NSG-E variant with:
• Mini PCIe WiFi Module:
– IG Qualcomm Atheros QCA9892-BR4B
– 2T2R / Wave 1 / Data rate up to 867Mbps
• Dual band support (2.4 or 5 GHz) (11b/g/n/ac). AP can work in either band but not both.
• Standard wireless security: WEP, WPA, WPA2, WPS
• Antenna options: Dual Band, Omni-Directional, External / Detachable
The WiFi feature supports:
• A maximum of 8 SSIDs per card.
• Public SSIDs
– Open or Pre-Shared Key (WPA/WPA2 for each SSID)
– Internal captive portal support to display Corporate Use Policy to connecting users
• Private SSIDs
– Pre-Shared Key (WPA/WPA2 for each SSID)
– L2/MAC Authentication
• Whitelist and Blacklist for UEs
• Sticky IP to allow same IP address for same MAC (e.g. for wireless printers)

9.2. WiFi on NSG 616

 VNS User Guide, Release 5.3.2

• Configuration by CSPRoot and Enterprise Administrator.

• All the capabilities that are available to hosts connected to a physical access port are also available to WiFi
connected hosts.
• WiFi Module Status (per Branch) and UE Status (per Branch, Domain, Enterprise)
• UE Statistics and Alarms
In order to support the WiFi card configuration, the following new constructs are introduced with corresponding
attributes:
• A new Access Port called WiFiPort. The new port enables WiFi card level attributes to be configured, specifi-
cally:
– Frequency Band
– Country Code
– WiFi Mode (a, b, g, n, ac)
– Frequency Channel
• SSIDConnection object to enable configuration of a single SSID on the WiFiPort (representing the WiFiCard).
The attributes include:
– SSID Name
– Authentication mode (e.g. Open, WEP, WPA, WPA1, WPA2)
– Passphrase
– Broadcast flag
– Custom properties (sent as a blob)
– A Whitelist and Blacklist of MAC Addresses
The new WiFiPort/SSID construct is logically equivalent to an Access port with an Untagged VLAN. The physical
WiFiPort is auto-named using the convention wlanX, where X is an integer. The new SSID Connection object enables
the configuration of multiple SSIDs (up to 8) on a single WiFiPort without overloading the VLAN attribute.
When the NSG is starting up, the OS will detect the installed WiFi module and load the appropriate drivers (CentOS
64 bit compatible drivers). The interface will not be usable until VSD has sent the configured information to the NSG.

Note: The WiFi interface is not required for bootstrapping. The configuration information is sent as part of the infra
configuration during bootstrapping or as a result of reload configuration post-bootstrapping. When an NSG is starting
up, the OS will detect the installed WiFi module and load the appropriate CentOS 64 compatible drivers. The WiFi
interface will not be usable until the VSD configuration is received.

9.2.2 Provisioning Workflow

The CSPRoot or Enterprise/Organization Administrator provisions at the NSG instance level since WiFi attributes are
locally unique.

Create WiFi Interface

Note that only a single WiFi port per NSG is supported.

9.2. WiFi on NSG 617

 VNS User Guide, Release 5.3.2

Fig. 9.24: Create WiFi Port (Basic and Advanced)

Step 1 From the Infrastructure view, navigate to the WiFi-capable NSG instance.
Step 2 In the NS Ports panel, select the WiFi Port icon at the bottom of the panel.
Step 3 In the New WiFi Port popup, enter a Name. (The physical name for the first created port is wlan0
and that physical name is read-only.) After completing configuration in the popup, the port will be
listed in the NS Ports panel as shown in the screenshot below captioned “Ports on NSG”. When the
WiFiPort is created, a default SSID connection is auto-created with default attributes.
Step 4 Enter a Description (optional). The Type is Access.
Step 5 In the Basic tab, enter WiFi Frequency Band, which must be in compliance with the regulations
of the country selected in the next step.
Step 6 To select the country, click the down arrow on the right of the Country field to display the list of
countries, and do any of the following:
• Select a country and hit Enter.
• To reduce the available options, start entering the country name, then hit Enter, select the coun-
try, and hit Enter again.
• Enter the country code instead of the country name, e.g., JP, US, CA, or GB and hit Enter.
Step 7 The country selection made in the previous step determines the selections available for Mode
and Frequency Channel. Choose the appropriate settings for these parameters, which must be in
compliance with the selected country’s regulations. If you make selections that are not supported,
an error message appears:

9.2. WiFi on NSG 618

 VNS User Guide, Release 5.3.2

Fig. 9.25: WiFi Deployment Not Supported in Selected Country

Step 8 In the Advanced tab, enter any additional custom properties. Click Create and the new WiFi port
appears in the list of NS Ports, as shown below in the screenshot captioned “Ports on NSG.”

Fig. 9.26: Ports on NSG

Create Captive Portal Profile

A Captive Portal Profile is required if the Captive Portal authentication mode is selected for an SSID connection (see
the Edit/Create SSID Connection (page 621) procedure below).
Step 1 In the Infrastructure view, from NSGs, navigate to Captive Portal Profiles and click the plus icon
to create a new profile.

9.2. WiFi on NSG 619

 VNS User Guide, Release 5.3.2

Step 2 Enter Name and optionally, Description. The Portal Type is Clickthrough.
Step 3 In the Captive Page field, enter the text of the organization’s Network Access Agreement (see
End User Workflow (page 624) below for a screenshot of an example agreement). No formatting is
necessary; however, basic HTML tagging such as bullet lists and bold and italic text is permissible
(as shown in the screenshot below, captioned “Captive Portal Profile With Page Content”). The
system will insert the text entered in this field into an HTML template. Images cannot be included
because security rules should prevent them from being called.

Fig. 9.27: Captive Portal Profile With Page Content

To see the end-user’s view of this text, click Preview. To complete the profile, click Create. The
new profile appears in the list of Captive Portal Profiles.

9.2. WiFi on NSG 620

 VNS User Guide, Release 5.3.2

Edit/Create SSID Connection

Fig. 9.28: Create SSID Connection

Step 1 Select the WiFi Port created in the Create WiFi Interface (page 617) procedure above. The SSID
panel appears to the right of the list of ports.
Step 2 Listed in the SSID panel is the first SSID, which is automatically created and has the default name
Nuage_Admin and an auto-generated passphrase. Nuage recommends that the name and password
of the default SSID be changed in accordance with your organization’s/enterprise’s policies.
• The default SSID may be edited but it cannot be deleted.
• By default, broadcast on this default SSID is disabled and we recommend that it remain so.
• VPorts are not supported on this interface.
• Up to seven additional SSIDs may be created. These additional SSIDs are virtual interfaces.
Step 3 To create a new SSID connection, in the SSID panel click the plus icon, and enter Name and
optionally, Description.
Step 4 Enable broadcast mode using the Broadcast SSID Enabled flag.
Step 5 On the Basic tab, specify Authentication Mode and Passphrase. Attributes on the Basic tab take
precedence if the same attributes are also included on the Advanced tab. For Authentication Mode,
the choices are:
• Open WiFi: For this authentication mode, Nuage recommends that:
– a separate subnet be configured
– PAT to underlay be enabled.
– Appropriate ACLs be defined so as not to block overlay connectivity. Any change to the
SSID is service-impacting to the UEs connected to the SSID.

9.2. WiFi on NSG 621

 VNS User Guide, Release 5.3.2

• WEP: For this authentication mode, all related attributes must be specified on the Advanced
tab.
• WPA
• WPA2
• WPA/WPA2 Mixed Mode
• Captive Portal: For this authentication mode, a Captive Portal Profile must be linked to the
SSID connection. (Instructions for creating this profile are in Create Captive Portal Profile
(page 619).) The mandatory Redirect Option parameter required by the Captive Portal authen-
tication mode can be either:
– Original Request: Select this to allow the end-user to access the URL he or she requested
before connecting to the SSID and the network access agreement was displayed, or
– Configure URL: Select this to have end-users redirected after they have accepted the terms
of the network access agreement. Enter the URL to which they are redirected in the Redi-
rect URL field, as shown below in the screenshot captioned “New URL for Captive Portal
Profile.”

Fig. 9.29: New URL for Captive Portal Profile

Step 6 On the Advanced tab, specify custom properties in the Generic Config field as a key=value pair.
Include on the Advanced tab those attributes that are not essential for beaconing (e.g. QoS, traffic
parameters). These will be pushed as a blob to the NSG. Examples: wpa_key_mgmt=WPA-PSK,
rsn_pairwise=CCMP.
Step 7 On the White/Black Listed Clients tab specify a list of MAC Addresses for white/black listing.

9.2. WiFi on NSG 622

 VNS User Guide, Release 5.3.2

Note: Blob attributes must be re-entered when NSG software is upgraded. The attributes will not
be retained across upgrades.

Attach Host/Bridge vPort

Fig. 9.30: Create vPort

Step 1 Navigate to a Domain -> Zone -> Subnet and attach a Bridge/Host vPort. Enter the following
information:
• For Network Services Gateways, select a WiFi-enabled NSG
• Select Interface Type as WiFi Interface
• For Port, select the WiFiPort
• For SSID, select an SSID
Step 2 Create a Bridge/Host vPort for each SSID (except for the default SSID).

Define DHCP Pool

Using the addresses in the DHCP Pool, the NSG will be able to service requests from the UEs and allocate a unique
IP addresses to all such host requests.
Step 1 Navigate to the Domain -> Zone -> Subnet for which the DHCP Address Pool is to be defined.

Step 2 Click on the icon on the far right. Add a new address range or edit an existing one.

9.2. WiFi on NSG 623

 VNS User Guide, Release 5.3.2

Fig. 9.31: Specify Address Range for DHCP Pool

Step 3 In the popup, specify Address Range by entering the First Address and the Last Address. In
addition, set DHCP Pool Type as Bridge or Host.

Note:
• If DHCP pool is exhausted, then a new client will get authenticated but will not get an IP
address.
• There is a default Inactivity Period of 15 mins, which is used to remove a UE if there is no
communication with an AP for that duration. Since the DHCP server will not release the IP until
lease expiry, the NSG will poll the connected MAC addresses every 15 minutes and release IPs
associated with unconnected MACs.

A DNS server may also be optionally configured using the available VSD configuration workflow.

End User Workflow

The end-user will connect to an SSID by selecting “Captive Portal”. The Network Access Agreement is displayed in
a browser window titled “WiFi Captive Portal” on the end-user’s device. The end-user must click the Accept button in
order to be authenticated, after which the device will be able to access network services.

9.2. WiFi on NSG 624

 VNS User Guide, Release 5.3.2

Fig. 9.32: WiFi Captive Portal

9.2.3 AP Management

AP management will include the following steps:

• Prepare: A WiFi integrated NSG is available for bootstrapping at a branch location.
• Setup: Define and configure SSIDs, DHCP Bridge Pool, type of authentication etc. for a branch. An enterprise
administrator will configure the necessary artefacts per the workflows described earlier.
• Deploy: Bootstrap the WiFi integrated NSG so that it can function as an Access Point.
• Manage: Change/maintain strong SSID password, add/edit SSIDs, manage whitelist, blacklist, etc.

Note: If an NSG is rebooted, UEs will reconnect when the NSG comes back up.

9.2.4 Statistics

If Stats have been enabled on the VSD, WiFi statistics are available at the Enterprise/Organization level, and accessed
as follows:

9.2. WiFi on NSG 625

 VNS User Guide, Release 5.3.2

Step 1 Navigate to Infrastructure -> NSG -> WiFi port and select the statistics icon . This opens a
new browser window for statistics visualization.
Step 2 Copy the ES Cluster Proxy Address from the Statistics section of the Platform System Configu-
ration (Settings -> System Configuration ).
Step 3 Paste the address:port number (6200) in another browser window and add the security exception
for certificates.
Step 4 Navigate back to the visualization window, and refresh it to view the WiFi statistics.

Fig. 9.33: WiFi Statistics

The graphic displays the number of Active SSIDs and Active Clients for the selected time interval,
and a list of the user end-points connected to the SSID. A user (identified by the MAC) may have
connected to an SSID multiple times and may have been assigned a different IP address each time.

9.2.5 Sample CLI Output

• get-ssids Displays all the SSIDs

• get-clients <ssid-interface> Displays information about clients connected to an SSID
• show-ports Displays all the resolved WiFi ports

9.2. WiFi on NSG 626

 VNS User Guide, Release 5.3.2

# nuage-nsg-wifi get-ssids

Interface wlan0.2
ifindex 48
wdev 0x6
addr 00:0e:8e:59:8e:fd
ssid mvnsgwifi01_1_2
type AP
channel 2 (2417 MHz), width: 20 MHz, center1: 2417 MHz
Interface wlan0.3
ifindex 47
wdev 0x5
addr 00:0e:8e:59:8e:fe
ssid mvnsgwifi01_1_3
type AP
channel 2 (2417 MHz), width: 20 MHz, center1: 2417 MHz
Interface wlan0.1
ifindex 45
wdev 0x3
addr 00:0e:8e:59:8e:fc
ssid mvnsgwifi01_1_1
type AP
channel 2 (2417 MHz), width: 20 MHz, center1: 2417 MHz
Interface wlan0
ifindex 8
wdev 0x1
addr 00:0e:8e:59:8e:fb
ssid Nuage_Admin
type AP
channel 2 (2417 MHz), width: 20 MHz, center1: 2417 MHz

# nuage-nsg-wifi get-clients wlan0.1

-----------------------------------------------
|Client | Mac | IP |
-----------------------------------------------
|1 | b8:27:eb:f3:87:5d | 10.10.10.46 |
|2 | b8:27:eb:6b:95:69 | 10.10.10.44 |
-----------------------------------------------

# nuage-nsg-wifi show-ports
Name: Host VPort bdce0bb7 UUID: 8fbf3336-bc23-4f05-bc5f-af3041dc6187
port-UUID: 8fbf3336-bc23-4f05-bc5f-af3041dc6187 Name: wlan0.2 MAC: b8:27:eb:
˓→5c:2e:38

Bridge: alubr0 port: 53 flags: 0x0 stats-interval: 60

vrf_id: 1066126378 evpn_id: 863055407 flow_flags: 0x21e64004 flood_
˓→gen_id: 0x1

IP: 10.10.10.15 subnet: 255.255.255.0 GW: 10.10.10.1

rate: 4294967295 kbit/s burst:4294967295 kB class:0 mac_count: 1
BUM rate: 4294967295 kbit/s BUM peak: 4294967295 kbit/s BUM burst:
˓→4294967295 kB

FIP rate: 4294967295 kbit/s FIP peak: 4294967295 kbit/s FIP burst:
˓→4294967295 kB

FIP Egress rate: 4294967295 kbit/s FIP Egress peak: 4294967295 kbit/s
˓→ FIP Egress burst: 4294967295 kB
Trusted: false Rewrite: false
RX packets:105 errors:0 dropped:0 rl_dropped:0
TX packets:306 errors:0 dropped:0

9.2. WiFi on NSG 627

 VNS User Guide, Release 5.3.2

RX bytes:28620 TX bytes:58974
anti-spoof: Disabled
policy group tags: 0x19c9 0x297d 0x5000 0x4000
policy group domain_id: 0xa1068
route_id: 0x62
class_id: 16(0x10)

Name: Bridge VPort 4a1e6549 UUID: 8ecf3c78-9ffc-48fd-9943-60e262b7cc4d

port-UUID: 8ecf3c78-9ffc-48fd-9943-60e262b7cc4d Name: wlan0.1 MAC: 00:00:00:
˓→00:00:00

Bridge: alubr0 port: 17 flags: 0x0 stats-interval: 60

vrf_id: 1066126378 evpn_id: 863055407 flow_flags: 0x31e04004 flood_
˓→gen_id: 0x1

rate: 4294967295 kbit/s burst:4294967295 kB class:0 mac_count: 2

BUM rate: 4294967295 kbit/s BUM peak: 4294967295 kbit/s BUM burst:
˓→4294967295 kB

FIP rate: 4294967295 kbit/s FIP peak: 4294967295 kbit/s FIP burst:
˓→4294967295 kB

FIP Egress rate: 4294967295 kbit/s FIP Egress peak: 4294967295 kbit/s
˓→ FIP Egress burst: 4294967295 kB
Trusted: false Rewrite: false
RX packets:28714 errors:0 dropped:0 rl_dropped:0
TX packets:29600 errors:0 dropped:0
RX bytes:2767185 TX bytes:6195167
anti-spoof: Disabled
policy group tags: 0x19c9 0x297d 0x5000 0x4000
policy group domain_id: 0xa1068
route_id: 0xa
class_id: 13(0xd)

Name: Bridge VPort 311015b6 UUID: a1430cdf-5fe1-4156-a20b-61e3b9897a02

port-UUID: a1430cdf-5fe1-4156-a20b-61e3b9897a02 Name: wlan0.3 MAC: 00:00:00:
˓→00:00:00

Bridge: alubr0 port: 52 flags: 0x0 stats-interval: 60

vrf_id: 1066126378 evpn_id: 863055407 flow_flags: 0x31e64004 flood_
˓→gen_id: 0x1

rate: 4294967295 kbit/s burst:4294967295 kB class:0 mac_count: 1

BUM rate: 4294967295 kbit/s BUM peak: 4294967295 kbit/s BUM burst:
˓→4294967295 kB

FIP rate: 4294967295 kbit/s FIP peak: 4294967295 kbit/s FIP burst:
˓→4294967295 kB

FIP Egress rate: 4294967295 kbit/s FIP Egress peak: 4294967295 kbit/s
˓→ FIP Egress burst: 4294967295 kB
Trusted: false Rewr ite: false
RX packets:235 error s:0 dropped:0 rl_dropped:0
TX packets:295 error s:0 dropped:0
RX bytes:29493 TX bytes:4266 2
anti-spoof: Disabled
policy group tags: 0 x19c9 0x297d 0x5000 0x4000
policy group domain_id: 0xa1068
route_id: 0x6d
class_id: 15(0xf)

9.2. WiFi on NSG 628

 CHAPTER

TEN

RESOURCES

• VSC Tools Commands (page 630)

629
 VNS User Guide, Release 5.3.2

10.1 VSC Tools Commands

• Overview (page 631)

• Show Commands (page 631)
– Help (page 631)
– OF-Status (page 632)
– Agent-Version (page 633)
– Agent-Status (page 634)
– Bridge-Flows (page 634)
– Service-Flows (page 635)
– Service-Details (page 637)
– Service-MAC-Table (page 640)
– FIB (page 641)
– Ports (page 642)
– Port-Stats (page 643)
– Service-Subnets (page 644)
– Service-Domains (page 645)
– Datapath-Flows (page 646)
– Datapath-Ports (page 647)
– Tunnel-Info (page 647)
– ACL (page 648)
– System-Info (page 650)
– QoS (page 651)
– Port-Status (page 651)
– System-Connections (page 651)
– NTPQ (page 652)
– IP-Table (page 652)
– IP-Table-NAT (page 653)
– Routing-Table (page 654)
• Debug Commands (page 654)
– Help (page 654)
– Tcpdump (page 655)
– Ethtool (page 656)
– Dig (page 656)
• Configure Commands (page 657)

10.1. VSC Tools Commands 630

 VNS User Guide, Release 5.3.2

– Help (page 657)

– Port-Up (page 658)
– Port-Down (page 658)
– Reboot (page 658)
– Shutdown (page 659)

10.1.1 Overview

This chapter describes the tools commands that can be executed from the VSC for monitoring and troubleshooting.
The commands are classified as show, debug and configure.
When a command is executed for a specific function, all the retrieved information is displayed. The options can be
utilized to filter the results (using data or service filters) or specify the output format (text, json or xml).
• Service filters: evpnid (-e) and vrfid (-d)
• Data Filters: ipaddr (-ip) , interface (-i), macaddr (-m), portname (-n) and vcid (-v)

Note: For readability, most of the examples in this chapter display only a representative sample output and not the
full set of retrieved information.

The output format options are supported for most functions except for the Linux based commands, which only support
text. The examples show JSON output for commands that support the format or text (which is the default output
format).

10.1.2 Show Commands

Help

The help argument can be used to list all the Show functions.
Example:

*A:SDN# tools vswitch <ip-address> command "show -h"

usage: show [-h] FUNCTION

optional arguments:
-h, --help show this help message and exit

FUNCTION:
For the specific FUNCTION options use 'show FUNCTION -h'

{of-status,agent-version,agent-status,bridge-flows,service-flows,service-
˓→details,service-mac- table,fib,ports,port-stats,host-ports,host-port-stats,bridge-
˓→ports,bridge-port-stats,service-subnets,service-domains,datapath-flows,datapath-

˓→ports,tunnel-info,acl,system-info,qos,port-status,system-connections,ntpq,ip-

˓→table,ip-table-nat,routing-table}

of-status Displays OpenFlow status (ovs-appctl ofproto/ show alubr0) and

˓→exit
agent-version Displays OVS version (ovs-vsctl -version) and exit

10.1. VSC Tools Commands 631

 VNS User Guide, Release 5.3.2

agent-status Displays OVS status (ovs-vsctl show) and exit

bridge-flows Display the bridge flows of the vswitch based on ovs-appctl
˓→bridge/dump-flows

service-flows List the flows specific to service based on ovs-appctl evpn/

˓→dump-flows

service-details List the service specific details like ports,acl and based on
˓→ovs-appctl host/port-show, bridge/port-showbridge/acl-able, evpn/mac-table

service-mac-table Displays the MAC table for the serv ice(s) and based on ovs-
˓→appctl evpn/mac-table

fib Displays FIB details of the service(s) and based on ovs-appctl

˓→evpn/dump-flows, ovs-appctl evpn/mac-table

ports List all the bridge and host ports of the service(s) and based
˓→on ovs-appctl host/port-show, bridge/port-show

port-stats List all the bridge and host port stats of the service(s) and
˓→based on ovs-appctl host/port-show,bridge/port-show

host-ports List all host ports of the service(s) and based on ovs-appctl
˓→host/port-show

host-port-stats List all the host port stats of the service(s) and based on ovs-
˓→appctl host/port-show

bridge-ports List all the bridge ports of the service(s) and based on ovs-
˓→appctl bridge/port-show

bridge-port-stats List all the bridge port stats of the service(s) and based on
˓→ovs-appctl bridge/port-show

service-subnets Display the service subnets details and based on ovs-appctl

˓→evpn/show

service-domains Display the service domains details and based on ovs-appctl vrf/
˓→show

datapath-flows List all the datapath dumpflows and based on ovs-dpctl dump-
˓→flows

datapath-ports List all the datapath ports and based on ovs-appctl dpif/show
tunnel-info List tunnel info(s) and based on ovs-ofctl dump-ports,ovs-
˓→appctl evpn/mac-table

acl List all the acl of the service(s) and based on ovs-appctl
˓→bridge/acl-table

system-info Executes linux command 'top -b -n 1'

qos Executes linux command 'tc -s qdisc show dev <interface>',By
˓→default it will use 'port1' as interface

port-status Executes linux command 'ip link show dev <interface>',By

˓→default it will use 'port1' as interface

system-connections Executes linux command 'ss'

ntpq Executes linux command 'ntpq -p'
ip-table Executes linux command 'iptables -L'
ip-table-nat Executes linux command 'iptables -t nat -L'
routing-table Executes linux command 'ip route show'

OF-Status

Syntax show of-status

Options No filters
Description Displays OpenFlow status.
Linux Command Context
ovs-appctl ofproto/show alubr0
Output Text (default), JSON, XML

10.1. VSC Tools Commands 632

 VNS User Guide, Release 5.3.2

Example

*A:SDN>tools>vswitch# command “show of-status -o json”

{
"stats_collectors":{
"state":"BACKOFF",
"tcp":"172.22.24.16:39090",
"16":"39090"
},
"controllers":{
"fail_mode":"secure",
"name":"ctrl1",
"local_ip":"10.10.10.119",
"probe":"1",
"tcp":"10.10.10.1:6633",
"NAT_T":"Disabled",
"state":"ACTIVE",
"role":"master",
"band":"out"
},
"BASIC_INFO":{
"Name":"alubr0",
"acl_udp_timeout":"180",
"gen_id":"0x2ff",
"acl_tcp_timeout":"180",
"mgmt":"0.0.0.0",
"dp_id":"0x1",
"personality":"vrs-g",
"vrs_role":"other",
"cleanup_pending":"0"
},
"dhcp-relay":{
"state":"Disabled",
"dhcp-sock":"-1"
}
}

Agent-Version

Syntax show agent-version

Options No filters
Description Displays Open vSwitch (OVS) version.
Linux Command Context
ovs-vsctl –version
Output Text
Example

*A:SDN# tools vswitch <ip-address> command "show agent-version"

ovs-vsctl (Open vSwitch) 3.2.5-143-nuage
Compiled Oct 15 2015 18:04:40
DB Schema 7.6.3

10.1. VSC Tools Commands 633

 VNS User Guide, Release 5.3.2

Agent-Status

Syntax show agent-status

Options No filters
Description Displays Open vSwitch (OVS) status.
Linux Command Context
ovs-vsctl show
Output Text
Example

*A:SDN# tools vswitch <ip-address> command "show agent-status"

fb50364b-428d-4f91-9333-6d85abc153ba
Bridge "alubr0"
Controller "ctrl1"
target: "ssl:10.10.13.7:6633"
role: slave
is_connected: true
Controller "ctrl2"
target: "ssl:10.10.15.9:6633"
role: master
is_connected: true
Port "port3.5"
Interface "port3.5"
Port "port3.7"
Interface "port3.7"
Port "svc-rl-tap1"
Interface "svc-rl-tap1"
Port "port3.1"
Interface "port3.1"
Port "alubr0"
Interface "alubr0"
type: internal
Port svc-pat-tap
Interface svc-pat-tap
type: internal
Port "port3.3"
Interface "port3.3"
Port "svc-rl-tap2"
Interface "svc-rl-tap2"
ovs_version: "3.2.5-143-nuage"

Bridge-Flows

Syntax show bridge-flows

Options -i or -m from data filters
Description Lists all flows in bridge
Linux Command Context
ovs-appctl bridge/dump-flows
Output Text (default), JSON, XML

10.1. VSC Tools Commands 634

 VNS User Guide, Release 5.3.2

Example

*A:SDN>tools>vswitch# command "show bridge-flows -o json”

[
{
"priority":"1",
"n_packets":"0",
"n_bytes":"0",
"duration":"4472524s",
"table_id":"",
"cookie":"0x2ff",
"nw_dst":"",
"nw_src":"",
"actions":"strip_vlan,resubmit(,22),resubmit(,23),move=NXM_OF_IN_PORT[]->NXM_
˓→NX_REG4[0..15],move:NXM_NX_REG1[0..15]->NXM_OF_IN_PORT[],resubmit(,4) ,move=NXM_OF_
˓→VLAN_TCI[0..11]->NXM_NX_REG1[0..11]"

},
{
"priority":"0",
"n_packets":"731682",
"n_bytes":"161913708",
"duration":"5569s",
"table_id":"4",
"cookie":"0x2ff",
"nw_dst":"",
"nw_src":"",
"actions":"resubmit(,7)"
}
]

Service-Flows

Syntax show service-flows

Options -d or -e from service filter and/or -i or -m or -n from data filter
Description Displays the flows for all services by default (if no filter specified).
Linux Command Context
ovs-appctl vrf/show alubr0
ovs-appctl evpn/list alubr0
ovs-appctl evpn/dump-flows alubr0 <EvpnId>
Output Text (default), JSON, XML
Example

*A:SDN>tools>vswitch# command "show service-flows -o json”

{
"VRF_LIST":{
"20001":{
"EVPN_LIST":{
"20003":{
"routes":[
{
"priority":"0",
"n_packets":"0",
"cookie":"0xc",

10.1. VSC Tools Commands 635

 VNS User Guide, Release 5.3.2

"evpn_id":"0x4e23",
"n_bytes":"0",
"duration":"516s",
"table_id":"11",
"vrf_id":"0x4e21",
"actions":"FLOOD"
}
],
"nw_dst":{
"10.39.76.2":{
"arp_routes":[
{
"priority":"32802",
"n_packets":"0",
"cookie":"0xc",
"evpn_id":"0x4e23",
"n_bytes":"0",
"duration":"515s",
"table_id":"12",
"vrf_id":"0x4e21",
"nw_dst":"10.39.76.2",
"actions":[
"resubmit(,11)",
{
"mod_dl_src":"68:54:ed:00:00:02",
"mod_dl_dst":"18:19:ab:b5:46:01"
}
]
}
],
"vrf_routes":[
{
"priority":"32802",
"n_packets":"0",
"cookie":"0xc",
"n_bytes":"0",
"duration":"515s",
"table_id":"13",
"vrf_id":"0x4e21",
"nw_dst":"10.39.76.2",
"actions":[
"resubmit(,12)",
"dec_ttl",
{
"set_evpn_id":"0x4e23"
}
]
}
]
},
"10.39.76.3":{
"arp_routes":[
{
"priority":"32802",
"n_packets":"0",
"cookie":"0xc",
"evpn_id":"0x4e23",
"n_bytes":"0",

10.1. VSC Tools Commands 636

 VNS User Guide, Release 5.3.2

"duration":"516s",
"table_id":"12",
"vrf_id":"0x4e21",
"nw_dst":"10.39.76.3",
"actions":[
"resubmit(,11)",
{
"mod_dl_src":"68:54:ed:00:00:02",
"mod_dl_dst":"18:cd:5c:b5:46:02"
}
]
}
],
"vrf_routes":[
{
"priority":"32802",
"n_packets":"0",
"cookie":"0xc",
"n_bytes":"0",
"duration":"516s",
"table_id":"13",
"vrf_id":"0x4e21",
"nw_dst":"10.39.76.3",
"actions":[
"resubmit(,12)",
"dec_ttl",
{
"set_evpn_id":"0x4e23"
}
]
}
]
}
}
},

}
}
}
}

Service-Details

Syntax show service-details

Options -d or -e from service filter and/or -i or -m or -n from data filter
Description List all service details, such as acls and ports.
Linux Command Context
ovs-appctl vrf/show alubr0
ovs-appctl evpn/list alubr0
ovs-appctl host/port-show
ovs-appctl bridge/port-show
ovs-appctl bridge/acl-table alubr0
ovs-appctl evpn/mac-table <evpn-id> alubr0

10.1. VSC Tools Commands 637

 VNS User Guide, Release 5.3.2

Output Text (default), JSON, XML

Example

*A:SDN>tools>vswitch# command “show service-details -o json”

{
"PORT_INFO":{
"vm-ports":[
{
"PORT TYPE":"VM",
"UUID":"1e679983-68e5-4b55-94a4-118791fcc60a",
"PORT ID":"1458",
"PORT NAME":"118-vm7",
"DISPLAY NAME":"118-vm7",
"MAC":"18:cd:5c:b5:46:07",
"IP":"10.31.122.3",
"VRF ID":"20001",
"EVPN ID":"20002",
"GATEWAY":"10.31.122.1"
}
],
"bridge-ports":[
{
"PORT TYPE":"Bridge",
"UUID":"ef80a514-04f3-4598-9465-38ebc70b4940",
"PORT ID":"9",
"PORT NAME":"eth1.105",
"DISPLAY NAME":"Bridge-210-eth1-105",
"MAC":"00:00:00:00:00:00",
"IP":"",
"VRF ID":"20001",
"EVPN ID":"20003",
"GATEWAY":""
}
],
"host-ports":[
{
"PORT TYPE":"Host",
"UUID":"5642ef12-f9db-46a0-ab05-50da399747bc",
"PORT ID":"6",
"PORT NAME":"eth1.102",
"DISPLAY NAME":"HOST-210-eth1-102",
"MAC":"1a:1a:1d:1d:1d:1d",
"IP":"0.0.0.0",
"VRF ID":"0",
"EVPN ID":"20021",
"GATEWAY":"0.0.0.0"
}
]
},
"TUNNEL_INFO":[
{
"tunnel id":"10001",
"Mac":[
"18:19:aa:b5:46:01"
],
"port_id":"1495",
"VM Port name":{

10.1. VSC Tools Commands 638

 VNS User Guide, Release 5.3.2

"type":"Vxlan",
"key(VC_ID)":"10001",
"NH_IP":"10.10.10.3"
}
}
],
"VRF_LIST":{
"20001":{
"EVPN_LIST":{
"20003":{
"gw":"10.39.76.1",
"vni_id":"0x2712",
"subnet":"10.39.76.0",
"mask":"255.255.255.0",
"evpn_dhcp_enabled":"DISABLED",
"evpn_id":"20003",
"ref_cnt":"3",
"ltep_port":"1493",
"evpn_arp_proxy":"DISABLED",
"cookie":"36175872",
"gen_id":"0xc",
"aging_period":"300",
"mac_count":"2",
"gw_mac":"68:54:ed:00:00:02",
"mode":"L3_MODE"
}
}
}
},
"PORT_ACL_LIST":{
"ingress":[
{
"PORT ID":"1458",
"PORT NAME":"118-vm7",
"MAC":"18:cd:5c:b5:46:07",
"IP":"10.31.122.3",
"ACL RULES":{
"priority":"0",
"in_port":"1458",
"actions":"acl_allow"
}
}
],
"egress":[
{
"PORT ID":"1458",
"PORT NAME":"118-vm7",
"MAC":"18:cd:5c:b5:46:07",
"IP":"10.31.122.3",
"ACL RULES":[
"ip",
{
"priority":"4",
"nw_src":"10.31.122.0/24",
"in_port":"1458",
"actions":"acl_allow"
}
]

10.1. VSC Tools Commands 639

 VNS User Guide, Release 5.3.2

}
],
"ingress-advanced":[
{
"PORT ID":"1458",
"PORT NAME":"118-vm7",
"MAC":"18:cd:5c:b5:46:07",
"IP":"10.31.122.3",
"ACL RULES":{
"priority":"0",
"in_port":"1458",
"actions":"acl_allow"
}
}
]
}
}

Service-MAC-Table

Syntax show service-mac-table

Options -d or -e from service filter and/or -m or -n from data filter
Description Displays the MAC table for the service(s)
Linux Command Context
ovs-appctl vrf/show alubr0
ovs-appctl evpn/list alubr0
ovs-appctl evpn/mac-table <evpn-id> alubr0
Output Text (default), JSON, XML
Example

*A:SDN>tools>vswitch# command “show service-mac-table -o json”

{
"VRF_LIST":{
"20001":{
"EVPN_LIST":{
"20003":{
"evpn_data":{
"gw":"10.39.76.1",
"vni_id":"0x2712",
"subnet":"10.39.76.0",
"mask":"255.255.255.0",
"evpn_dhcp_enabled":"DISABLED",
"evpn_id":"20003",
"ref_cnt":"3",
"ltep_port":"1493",
"evpn_arp_proxy":"DISABLED",
"cookie":"36175872",
"gen_id":"0xc",
"aging_period":"300",
"mac_count":"2",
"gw_mac":"68:54:ed:00:00:02",

10.1. VSC Tools Commands 640

 VNS User Guide, Release 5.3.2

"mode":"L3_MODE"
},
"mac_table":[
{
"18:cd:5c:b5:46:02":{
"Mac":"18:cd:5c:b5:46:02",
"Port":"1463",
"Duration":"389s",
"Expiry":"0s",
"Cookie":"0xc",
"Pkt Count":"0",
"Pkt Bytes":"0",
"VM Port name":"118-vm2 (118-vm2)"
}
}
]
}

}
}
}
}

FIB

Syntax show fib

Options -d or -e from service filter and/or -i or -m or -n from data filter
Description Displays the FIB information including MAC and IP addresses for hosts attached to NSG
access ports.
Linux Command Context
ovs-appctl vrf/show alubr0
ovs-appctl evpn/list alubr0
ovs-appctl evpn/dump-flows alubr0 <evpn-id>
ovs-appctl evpn/mac-table <evpn-id> alubr0
Output Text (default), JSON, XML
Example

*A:SDN>tools>vswitch# command “show fib -o json”

[
{
"VRF ID":"20001",
"EVPN ID":"20003",
"IP":"10.39.76.2",
"MAC":"18:19:ab:b5:46:01",
"PORT":"1496",
"PORT NAME":{
"type":"Vxlan",
"key(VC_ID)":"10002",
"NH_IP":"10.10.10.3"
}
},
{

10.1. VSC Tools Commands 641

 VNS User Guide, Release 5.3.2

"VRF ID":"20001",
"EVPN ID":"20003",
"IP":"10.39.76.3",
"MAC":"18:cd:5c:b5:46:02",
"PORT":"1463",
"PORT NAME":"118-vm2 (118-vm2)"
},
]

Ports

Syntax show ports

Options -d or -e from service filter and/or -i or -m or -n from data filter
Description Show ports returns combined results of vPorts type Bridge/Host. Individual functions
(bridge-ports, host-ports) can be used to display results for the specific vPort type.
Linux Command Context
ovs-appctl host/port-show
ovs-appctl bridge/port-show
Output Text (default), JSON, XML
Example

*A:SDN>tools>vswitch# command "show ports -o json”

{
"vm-ports":[
{
"PORT TYPE":"VM",
"UUID":"1e679983-68e5-4b55-94a4-118791fcc60a",
"PORT ID":"1458",
"PORT NAME":"118-vm7",
"DISPLAY NAME":"118-vm7",
"MAC":"18:cd:5c:b5:46:07",
"IP":"10.31.122.3",
"VRF ID":"20001",
"EVPN ID":"20002",
"GATEWAY":"10.31.122.1"
}
],
"bridge-ports":[
{
"PORT TYPE":"Bridge",
"UUID":"ef80a514-04f3-4598-9465-38ebc70b4940",
"PORT ID":"9",
"PORT NAME":"eth1.105",
"DISPLAY NAME":"Bridge-210-eth1-105",
"MAC":"00:00:00:00:00:00",
"IP":"",
"VRF ID":"20001",
"EVPN ID":"20003",
"GATEWAY":""
}
],

10.1. VSC Tools Commands 642

 VNS User Guide, Release 5.3.2

"host-ports":[
{
"PORT TYPE":"Host",
"UUID":"5642ef12-f9db-46a0-ab05-50da399747bc",
"PORT ID":"6",
"PORT NAME":"eth1.102",
"DISPLAY NAME":"HOST-210-eth1-102",
"MAC":"1a:1a:1d:1d:1d:1d",
"IP":"0.0.0.0",
"VRF ID":"0",
"EVPN ID":"20021",
"GATEWAY":"0.0.0.0"
}
]
}

Port-Stats

Syntax show port-stats

Options -d or -e from service filter and/or -i or -m or -n from data filter
Description Show port-stats returns combined results of vPorts type Bridge/Host. Individual functions
(bridge-ports, host-ports) can be used to display results for the specific vPort type.
Linux Command Context
ovs-ofctl dump-ports alubr0
ovs-appctl host/port-show
ovs-appctl bridge/port-show
Output Text (default), JSON, XML
Example

*A:SDN>tools>vswitch# command "show port-stats -o json”

[
{
"PORT ID":"11",
"TYPE":"Bridge",
"PORT NAME":"eth2.100",
"MAC":"00:00:00:00:00:00",
"Rx Pkts":"0",
"Rx Bytes":"0",
"Rx Drop":"0",
"Rx Errs":"0",
"Rx frame":"0",
"Rx Over":"0",
"Rx CRC":"0",
"Tx Pkts":"0",
"Tx Bytes":"0",
"Tx Drop":"0",
"Tx Errs":"0",
"Tx coll":"0"
},
{
"PORT ID":"13",
"TYPE":"Bridge",

10.1. VSC Tools Commands 643

 VNS User Guide, Release 5.3.2

"PORT NAME":"eth2.201",
"MAC":"00:00:00:00:00:00",
"Rx Pkts":"0",
"Rx Bytes":"0",
"Rx Drop":"0",
"Rx Errs":"0",
"Rx frame":"0",
"Rx Over":"0",
"Rx CRC":"0",
"Tx Pkts":"0",
"Tx Bytes":"0",
"Tx Drop":"0",
"Tx Errs":"0",
"Tx coll":"0"
}
]

Service-Subnets

Syntax show service-subnets

Options -e from service filter and/or -i or -m or -n from data filter
Description Display subnet details of the service
Linux Command Context
ovs-appctl evpn/show alubr0
Output Text (default), JSON, XML
Example

*A:SDN>tools>vswitch# command “show service-subnets -o json”

{
"EVPN_LIST":{
"20003":{
"evpn_data":{
"gw":"10.100.10.1",
"mac_count":"0",
"subnet":"10.100.10.0",
"ref_cnt":"2",
"dhcp_relay":"DISABLED",
"cookie":"640155648",
"gw_mac":"68:54:ed:00:00:01",
"gen_id":"0x2",
"vni_id":"0x2",
"arp_proxy":"DISABLED",
"mask":"255.255.255.0",
"evpn_id":"20003",
"dhcp_enabled":"ENABLED",
"ltep_port":"10",
"mode":"L3_MODE",
"aging_period":"300"
},
"table_data":[
{
"PORT NAME":"eth1.105",

10.1. VSC Tools Commands 644

 VNS User Guide, Release 5.3.2

"PORT ID":"9",
"Ref Count":"14",
"Flood":"yes",
"IP":"0.0.0.0",
"SUBNET":"255.255.255.0",
"Gateway":"10.100.10.1"
},
{
"PORT NAME":"ltep-2",
"PORT ID":"10",
"Ref Count":"0",
"Flood":"no",
"IP":"",
"SUBNET":"",
"Gateway":""
}
]
}
}
}

Service-Domains

Syntax show service-domain

Options -d from service filter and/or -i or -m or -n from data filter
Description Display domain details of the service
Linux Command Context
ovs-appctl vrf/show alubr0
Output Text (default), JSON, XML
Example

*A:SDN>tools>vswitch# command “show service-domains -o json”

{
"VRF_LIST":{
"20001":{
"vrf_data":{
"ref_cnt":"2",
"gen_id":"0xc",
"ltep_port":"1491",
"mvpn_id":"0x4801",
"evpn-list":"20002 20003",
"vrf_id":"0x4e21"
},
"table_data":[
{
"PORT NAME":"taaa34801",
"PORT ID":"1494",
"Ref Count":"4",
"Flood":"no",
"IP":"",
"SUBNET":"",
"Gateway":""

10.1. VSC Tools Commands 645

 VNS User Guide, Release 5.3.2

},
{
"PORT NAME":"ltep-4801",
"PORT ID":"1491",
"Ref Count":"0",
"Flood":"no",
"IP":"",
"SUBNET":"",
"Gateway":""
}
]
}
}
}

Datapath-Flows

Syntax show datapath-flows

Options -i or -m from data filter
Description Display all the datapath dumpflows
Linux Command Context
ovs-dpctl dump-flows
Output Text (default), JSON, XML
Example

*A:SDN>tools>vswitch# command "show datapath-flows -o json”

[
{
"IN PORT":"1496",
"OUT PORT":"1463",
"IPv4 ":{
"frag":"no",
"src":"10.31.122.2",
"ttl":"63",
"tos":"0",
"dst":"10.39.76.3",
"proto":"1"
},
"ETH":{
"src":"68:54:ed:00:00:02",
"dst":"18:cd:5c:b5:46:02"
},
"ETH TYPE":"0x0800",
"SET":{
"outer_tos":"0"
}
}
]

10.1. VSC Tools Commands 646

 VNS User Guide, Release 5.3.2

Datapath-Ports

Syntax show datapath-ports

Options -i or -n from data filter
Description Display name and type of all the datapath ports
Linux Command Context
ovs-appctl dpif/show
Output Text (default), JSON, XML
Example

*A:SDN>tools>vswitch# command "show datapath-ports -o json”

{
"alubr0":[
{
"NAME":"alubr0 65534/2",
"TYPE":" (internal)"
},
{
"NAME":"svc-pat-tap 1/1",
"TYPE":" (internal)"
},
{
"NAME":"svc-rl-tap1 2/3",
"TYPE":" (system)"
},
{
"NAME":"svc-rl-tap2 3/4",
"TYPE":" (system)"
}
]
}

Tunnel-Info

Syntax show tunnel-info

Options -v from data filter
Description Displays the tunnel endpoint details including statistics
Linux Command Context
ovs-appctl vrf/show alubr0
ovs-appctl evpn/list alubr0
ovs-ofctl dump-ports alubr0
ovs-appctl evpn/mac-table <evpn-id> alubr0
Output Text (default), JSON, XML
Example

10.1. VSC Tools Commands 647

 VNS User Guide, Release 5.3.2

*A:SDN>tools>vswitch# command "show tunnel-info -o json”

[
{
"TUNNEL ID":"10001",
"PORT ID":"1495",
"PORT NAME":{
"type":"Vxlan",
"key(VC_ID)":"10001",
"NH_IP":"10.10.10.3"
},
"MAC":[
"18:19:aa:b5:46:01"
],
"Rx Pkts":"0",
"Rx Bytes":"0",
"Rx Drop":"0",
"Rx Errs":"0",
"Rx frame":"0",
"Rx Over":"0",
"Rx CRC":"0",
"Tx Pkts":"0",
"Tx Bytes":"0",
"Tx Drop":"0",
"Tx Errs":"0",
"Tx coll":"0"
}
]

ACL

Syntax show acl

Options -d or -e from service filter and/or -i, -m, or -n from data filter
Description List all the acls of the service(s)
Linux Command Context
ovs-appctl bridge/acl-table alubr0
ovs-appctl host/port-show
ovs-appctl bridge/port-show
Output Text (default), JSON, XML
Example

*A:SDN>tools>vswitch# command "show acl -o json”

{
"ingress":[
{
"PORT ID":"1463",
"PORT NAME":"118-vm2",
"MAC":"18:cd:5c:b5:46:02",
"IP":"10.39.76.3",
"ACL RULES":[

10.1. VSC Tools Commands 648

 VNS User Guide, Release 5.3.2

"arp",
{
"priority":"1",
"in_port":"1463",
"actions":"acl_allow"
}
]
},
{
"PORT ID":"1463",
"PORT NAME":"118-vm2",
"MAC":"18:cd:5c:b5:46:02",
"IP":"10.39.76.3",
"ACL RULES":[
"ip",
{
"priority":"2",
"in_port":"1463",
"actions":"acl_allow"
}
]
}
],
"egress":[
{
"PORT ID":"1458",
"PORT NAME":"118-vm7",
"MAC":"18:cd:5c:b5:46:07",
"IP":"10.31.122.3",
"ACL RULES":[
"ip",
{
"priority":"4",
"nw_src":"10.31.122.0/24",
"in_port":"1458",
"actions":"acl_allow"
}
]
},
{
"PORT ID":"1458",
"PORT NAME":"118-vm7",
"MAC":"18:cd:5c:b5:46:07",
"IP":"10.31.122.3",
"ACL RULES":[
"ip",
{
"priority":"3",
"nw_src":"10.39.76.0/24",
"in_port":"1458",
"actions":"acl_allow"
}
]
}
],
"ingress-advanced":[
{
"PORT ID":"1458",

10.1. VSC Tools Commands 649

 VNS User Guide, Release 5.3.2

"PORT NAME":"118-vm7",
"MAC":"18:cd:5c:b5:46:07",
"IP":"10.31.122.3",
"ACL RULES":{
"priority":"0",
"in_port":"1458",
"actions":"acl_allow"
}
},
{
"PORT ID":"1463",
"PORT NAME":"118-vm2",
"MAC":"18:cd:5c:b5:46:02",
"IP":"10.39.76.3",
"ACL RULES":{
"priority":"0",
"in_port":"1463",
"actions":"acl_allow"
}
}
]
}

System-Info

Syntax show system-info

Options No filters
Description Displays system information for the target
Linux Command Context
top -n 1
Output Text
Example

*A:SDN>tools>vswitch# command “show system-info”

top - 13:20:16 up 18 days, 21:31, 4 users, load average: 0.14, 0.05, 0.05
Tasks: 232 total, 1 running, 231 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.1 us, 0.2 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 32755200 total, 2613356 used, 30141844 free, 884 buffers
KiB Swap: 16523260 total, 0 used, 16523260 free. 1646724 cached Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

1 root 20 0 53964 7912 2548 S 0.0 0.0 0:43.10 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.82 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:01.78 ksoftirqd/0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:+
7 root rt 0 0 0 0 S 0.0 0.0 0:01.32 migration/0
8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
9 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/0
10 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/1
11 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/2
12 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/3
13 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/4

10.1. VSC Tools Commands 650

 VNS User Guide, Release 5.3.2

14 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/5

15 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/6
16 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/7
17 root 20 0 0 0 0 S 0.0 0.0 20:01.05 rcu_sched
18 root 20 0 0 0 0 S 0.0 0.0 3:50.38 rcuos/0

QoS

Syntax show qos -i port2

Options -i (interface) filter
Description Displays queuing discpline information for the interface
Linux Command Context
tc -s qdisc show dev <interface> (Note: default interface is port1)
Output Text
Example

*A:SDN>tools>vswitch# command “show qos -i port2”

qdisc pfifo_fast 0: root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1

Sent 127166604 bytes 1300512 pkt (dropped 0, overlimits 0 requeues 83)
backlog 0b 0p requeues 83

Port-Status

Syntax show port-status -i port2

Options -i (interface) filter
Description Display state of the interface
Linux Command Context
ip link show dev <interface> (Note: default interface is port1)
Output Text
Example

*A:SDN>tools>vswitch# command “show port-status -i port2”

2: port2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode

˓→DEFAULT qlen 1000

link/ether 00:25:90:de:98:38 brd ff:ff:ff:ff:ff:ff

System-Connections

Syntax show system-connections

Options No filters
Description Display socket statistics
Linux Command Context

10.1. VSC Tools Commands 651

 VNS User Guide, Release 5.3.2

ss
Output Text
Example

*A:SDN>tools>vswitch# command “show system-connections”

State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 0 135.227.229.210:ssh 172.22.24.57:53723
CLOSE-WAIT 1 0 135.227.229.210:58380 23.200.86.177:http
ESTAB 0 0 10.10.30.1:43537 50.10.10.2:6633
CLOSE-WAIT 1 0 135.227.229.210:58381 23.200.86.177:http
ESTAB 0 0 10.10.30.1:7406 50.10.10.2:50944
CLOSE-WAIT 1 0 135.227.229.210:55460 172.22.24.58:ssh
ESTAB 0 0 135.227.229.210:5901 135.227.229.28:57208
ESTAB 0 0 135.227.229.210:43303 135.227.229.246:29090

NTPQ

Syntax show ntpq

Options No filters
Description Display NTP peer information
Linux Command Context
ntpq -p
Output Text
Example

*A:SDN>tools>vswitch# command “show ntpq”

remote refid st t when poll reach delay offset jitter
==============================================================================
172.22.24.217 .INIT. 16 u - 1024 0 0.000 0.000 0.000

IP-Table

Syntax show ip-table

Options No filters
Description List all rules in iptable
Linux Command Context
iptables -L
Output Text
Example

*A:SDN>tools>vswitch# command “show ip-table”

Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT udp -- anywhere anywhere udp dpt:domain
ACCEPT tcp -- anywhere anywhere tcp dpt:domain
ACCEPT udp -- anywhere anywhere udp dpt:bootps
ACCEPT tcp -- anywhere anywhere tcp dpt:bootps

10.1. VSC Tools Commands 652

 VNS User Guide, Release 5.3.2

Chain FORWARD (policy ACCEPT)

target prot opt source destination
ACCEPT all -- anywhere 192.168.122.0/24 state RELATED,ESTABLISHED
ACCEPT all -- 192.168.122.0/24 anywhere
ACCEPT all -- anywhere anywhere
REJECT all -- anywhere anywhere reject-with icmp-port-
˓→unreachable

REJECT all -- anywhere anywhere reject-with icmp-port-

˓→unreachable

Chain OUTPUT (policy ACCEPT)

target prot opt source destination

IP-Table-NAT

Syntax show ip-table-nat

Options No filters
Description List all rules in iptable including NAT
Linux Command Context
iptables -t nat -L
Output Text
Example

*A:SDN>tools>vswitch# command “show ip-table-nat”

Chain PREROUTING (policy ACCEPT)
target prot opt source destination
PREROUTING_direct all -- anywhere anywhere
PREROUTING_ZONES_SOURCE all -- anywhere anywhere
PREROUTING_ZONES all -- anywhere anywhere

Chain INPUT (policy ACCEPT)

target prot opt source destination

Chain OUTPUT (policy ACCEPT)

target prot opt source destination
OUTPUT_direct all -- anywhere anywhere

Chain POSTROUTING (policy ACCEPT)

target prot opt source destination
MASQUERADE tcp -- 192.168.122.0/24 !192.168.122.0/24 masq ports: 1024-65535
MASQUERADE udp -- 192.168.122.0/24 !192.168.122.0/24 masq ports: 1024-65535
MASQUERADE all -- 192.168.122.0/24 !192.168.122.0/24
POSTROUTING_direct all -- anywhere anywhere
POSTROUTING_ZONES_SOURCE all -- anywhere anywhere
POSTROUTING_ZONES all -- anywhere anywhere

Chain OUTPUT_direct (1 references)

target prot opt source destination

Chain POSTROUTING_ZONES (1 references)

target prot opt source destination
POST_public all -- anywhere anywhere [goto]

10.1. VSC Tools Commands 653

 VNS User Guide, Release 5.3.2

POST_public all -- anywhere anywhere [goto]

POST_public all -- anywhere anywhere [goto]
POST_public all -- anywhere anywhere [goto]

Chain POSTROUTING_ZONES_SOURCE (1 references)

target prot opt source destination

Chain POSTROUTING_direct (1 references)

target prot opt source destination

Chain POST_public (4 references)

target prot opt source destination
POST_public_log all -- anywhere anywhere
POST_public_deny all -- anywhere anywhere
POST_public_allow all -- anywhere anywhere

Chain POST_public_allow (1 references)

target prot opt source destination

Chain POST_public_deny (1 references)

target prot opt source destination

Routing-Table

Syntax show routing-table

Options No filters
Description Displays content of routing table
Linux Command Context
ip route show
Output Text
Example

*A:SDN>tools>vswitch# command “show routing-table”

135.227.229.0/24 dev eth0 proto kernel scope link src 135.227.229.210

10.10.30.0/24 dev eth1 proto kernel scope link src 10.10.30.1
10.10.10.0/24 dev eth2 proto kernel scope link src 10.10.10.1
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
50.10.0.0/16 via 10.10.30.2 dev eth1
169.254.0.0/16 dev eth0 scope link metric 1002
169.254.0.0/16 dev eth1 scope link metric 1003
169.254.0.0/16 dev eth2 scope link metric 1004
default via 135.227.229.1 dev eth0

10.1.3 Debug Commands

Help

The help argument can be used to list all the Debug functions.
Example:

10.1. VSC Tools Commands 654

 VNS User Guide, Release 5.3.2

*A:SDN# tools vswitch <ip-address> command "debug -h"

usage: debug [-h] FUNCTION

optional arguments:
-h, --help show this help message and exit

FUNCTION:
For the specific FUNCTION options use 'debug FUNCTION -h'

{tcpdump,ethtool,dig}

tcpdump Executes linux command 'tcpdump -i <interface> -c <count> -vvv -n host

˓→<host> and port <port> and <proto>'. By default it will use 'port1' as interface
˓→and count to 100 packets

ethtool Executes linux command 'ethtool <interface>'. By default it will use

˓→'port1' as interface

dig Executes linux command 'dig <FQDN>'

Tcpdump

Syntax debug tcpdump -i port2

Options –i (interface),-c (packet count),-host (host),-port (port number) and -proto (Protocol) filters
Description Display network traffic
Linux Command Context
tcpdump -i <interface> -c <count> -vvv -n host <host> and port <port> and <proto> (Note: default
interface is port1)
Output Text
Example

*A:SDN>tools>vswitch# command "debug tcpdump -i port2 -c 50 -host a.b.c.d -port 1234

˓→ -proto tcp"

tcpdump: listening on port2, link-type EN10MB (Ethernet), capture size 65535 bytes
22:54:52.768679 IP (tos 0x0, ttl 64, id 23941, offset 0, flags [DF], proto TCP (6),
˓→length 244)

nsg-131-108-32-177.58637 > 50-10-10-2.gar.clearwire-wmx.net.7407: Flags [P.], cksum

˓→0x5afd

(incorrect -> 0x042c), seq 921194524:921194716, ack 2901986776, win 331, options
˓→[nop,nop,TS val

2695024815 ecr 5708294], length 192

22:54:52.769952 IP (tos 0xc0, ttl 63, id 39535, offset 0, flags [none], proto TCP
˓→(6), length 52)

50-10-10-2.gar.clearwire-wmx.net.7407 > nsg-131-108-32-177.58637: Flags [.], cksum

˓→0x6d1e

(correct), ack 192, win 8005, options [nop,nop,TS val 5708295 ecr 2695024815], length
˓→0

22:54:53.210682 IP (tos 0xc0, ttl 1, id 55413, offset 0, flags [none], proto OSPF
˓→(89), length 64 )

10.10.20.2 > ospf-all.mcast.net: OSPFv2, Hello, length 44

Router-ID 35.227.229.127, Backbone Area, Authentication Type: none (0)
Options [External]
Hello Timer 10s, D ead Timer 40s, Mask 255.255.255.0, Priority 1
Designated Router 10.10.20.2

10.1. VSC Tools Commands 655

 VNS User Guide, Release 5.3.2

........
......
nsg-131- 108-32-177.58637 > 50-10-10-2.gar.clearwire-wmx.net.7407: Flags [P.], cksum
˓→0x5b0a

(incorrect -> 0xa6c0), seq 229711:229916, ack 71, win 331, options [nop,nop,TS val
˓→2695031 006 ecr

5708307] , length 205

50 packets captured
64 packets received by filter
0 packets dropped by kernel

Ethtool

Syntax debug ethtool -i port2

Options -i (interface) filter
Description Display Ethernet device driver and hardware settings
Linux Command Context
ethtool <interface> (Note: default interface is port1)
Output Text
Example

*A:SDN>tools>vswitch# command "debug ethtool -i port2"

Settings for port2:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Speed: 100Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
MDI-X: off (auto)
Supports Wake-on: pumbg
Wake-on: g
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes

Dig

Syntax dig

10.1. VSC Tools Commands 656

 VNS User Guide, Release 5.3.2

Options No filters
Description DNS lookup utility
Linux Command Context
dig <FQDN>
Output Text
Example

*A:SDN>tools>vswitch# command "dig www.google.com"

; <<>> DiG 9.9.4-RedHat-9.9.4-14.el7_0.1 <<>> www.google.com

;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: FORMERR, id: 20000
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;www.google.com. IN A

;; Query time: 0 msec

;; SERVER: 128.251.10.29#53(128.251.10.29)
;; WHEN: Tue Sep 22 15:00:12 PDT 2015
;; MSG SIZE rcvd: 43

10.1.4 Configure Commands

Help

The help argument can be used to list all the Show functions.
Example:

*A:SDN# tools vswitch <ip-address> command "configure -h"

usage: configure [-h] FUNCTION

optional arguments:
-h, --help show this help message and exit

FUNCTION:
For the specific FUNCTION options use 'configure FUNCTION -h'

{port-up,port-down,reboot,shutdown}

port-up Executes linux command 'ip link set dev <interface> up'. By default it
˓→will use 'port1' as interface

port-down Executes linux command 'ip link set dev <interface> down'. By default it
˓→will use 'port1' as interface

reboot Executes linux command 'Shutdown -r now'

shutdown Executes linux command 'Shutdown -h now'

10.1. VSC Tools Commands 657

 VNS User Guide, Release 5.3.2

Port-Up

Syntax Configure port-up

Options -i (interface) filter
Description Change state of the interface to UP
Linux Command Context
ip link set dev <interface> up (Note: default interface is port1)
Output Text
Example

*A:SDN>tools>vswitch# command "configure port-up -i port2"

A:SDN>tools>vswitch#

Port-Down

Syntax Configure port-down

Options -i (interface) filter
Description Change state of the interface to DOWN
Linux Command Context
ip link set dev <interface> down (Note: default interface is port1)
Output Text
Example

*A:SDN>tools>vswitch# command "configure port-down -i port2"

A:SDN>tools>vswitch#

Reboot

Syntax Configure reboot

Options No filters
Description Reboot
Linux Command Context
shutdown -r now
Output Text
Example

*A:SDN>tools>vswitch# command "configure reboot"

A:SDN>tools>vswitch#

10.1. VSC Tools Commands 658

 VNS User Guide, Release 5.3.2

Shutdown

Syntax Configure shutdown

Options No filters
Description Shutdown
Linux Command Context
shutdown -h now
Output Text
Example

*A:SDN>tools>vswitch# command "configure shutdown"

A:SDN>tools>vswitch#

10.1. VSC Tools Commands 659