#CiscoLiveAPJC

-
 ACI – The Foundation of an Internal
Private Cloud
(aka “not just another network…”)
BRKDCN-2984

Steve Sharman – Solutions Engineer

BRKDCN-2984

-
#CiscoLiveAPJC
 https://ciscolive.ciscoevents.com/

Cisco Webex App ciscolivebot/#BRKDCN-2984

Questions?
Use Cisco Webex App to chat
with the speaker after the session

How
1 Find this session in the Cisco Live Mobile App

2 Click “Join the Discussion”

3 Install the Webex App or go directly to the Webex space

4 Enter messages/questions in the Webex space

Webex spaces will be moderated

by the speaker until November 15, 2024.

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
-
 At little bit of background to this session…

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
-
 Agenda

• Setting the scene

• Converting your fabric into Application Centric “mode”
• Working with ESGs
• Understanding ACI Security
• External Connectivity
• Increasing Security
• Automation Blueprints

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
-
 Before we get
started…

-
 Icons
L3out EPG ESG Cont
Tenant L3out EPG ESG Contract

Path C CCI I P C CCI I P

EPG ESG Subj
VRF Path Subject
EPG ESG
C CCI I P C CCI I P
C CCI I P
AP extEPG
Filt
App Profile External EPG
C CCI I P
Filter

BD
Entry
Bridge Domain
Entry
Subnets

Subnets *arrows indicate expected direction of connection i.e. from consumer to provider

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
-
 You are going to see lots (and lots) of diagrams…

BD 192.168.150.0_24 BD 192.168.151.0_24 BD 192.168.152.0_24

AP
EPG EPG EPG

Network-
segments
192.168.150.0_24 192.168.151.0_24 192.168.152.0_24
dynamic (P,S) vlans dynamic (P,S) vlans dynamic (P,S) vlans
C CCI I P C CCI I P C CCI I P

The details are there for your

reference so that you can rebuild
in your own environment

BD 192.168.150.0_24 BD 192.168.151.0_24 BD 192.168.152.0_24

AP
EPG EPG EPG
Network-
segments
192.168.150.0_24 192.168.151.0_24 192.168.152.0_24
dynamic (P,S) vlans dynamic (P,S) vlans dynamic (P,S) vlans

C CCI I P C CCI I P C CCI I P

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
-
 Setting the scene…
Designing your Tenants…
Switching to Application Centric mode…
Working with ESGs…
Understanding ACI security…
External Connectivity…
Increasing Security…
Automation Blueprints…
Wrapping up…

-
 Public Cloud
infrastructure…

-
 AWS reference architecture
https://docs.aws.amazon.com/vpc/latest/userguide/extend-intro.html

AWS Backbone

eu-west-1 Internet gateway eu-west-2 Internet gateway

eu-west-1-production eu-west-2-production
Public subnet Private subnet Public subnet Private subnet

NAT gateway Route table Route table Transit NAT gateway Route table Route table
Gateway

Customer Site Customer Site

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
-
 Network Connectivity and Security are mandatory in
the cloud…

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
-
 Different clouds run different hypervisors

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
-
 Executive Cross Functional
Sponsorship Teams

New Talent
Scaling
Attraction

New Culture Think Agile

Evolution Instead
Partnerships 2.0
of Revolution

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
-
 A cloud operating model succeeds best when there is a
new organizational culture…

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
-
 Cloud operating models have changed the way that
security is implemented…

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
-
 With a cloud operating model, security rules are typically
declared with the application constructs…

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
-
 Conversely, within enterprise Data Centers security has
been implemented by network and/or security
administrators at a VRF boundary…

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
-
 Traditional Enterprise Security Model

Outside Inside

ubuntu-01 ubuntu-02

permit ubuntu-01 ubuntu-02 tcp 5201

Traffic is routed through a firewall which

typically becomes a pinch point with
thousands of rules

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
-
 What are the network characteristics required to enable
us to operate in a cloud like manner…?

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
-
 ACI is the foundation for an internal private cloud…!

Day0 automation out-of-the- Pervasive

Per-application
box; physical fabric and Security Model
service-chaining
underlay

Hybrid cloud capability; Single API Model for 100s of Infrastructure as Code with
public cloud-like networking switches and 1000s of ports; Ansible and Terraform
constructs cloud-like consumption
model

Automation | Classification and Segmentation | Security

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
-
 The ACI Vision…

-
 The ACI reference application from circa 2014…

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
-
 The mythical three tier application…!
Web App DB

QoS QoS QoS

Outside
(Tenant VRF) Filter Service Filter

APIC

ACI Fabric Application Policy

Infrastructure
Controller

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
-
 Our reference application for this presentation…

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
-
 Online Boutique
https://github.com/GoogleCloudPlatform/microservices-demo

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
-
 Online Boutique
https://github.com/GoogleCloudPlatform/microservices-demo

Source/Consumer Target/Provider Target/Provider

frontend checkout Port
cart Redis cache TCP 6379

checkout cart TCP 7070

currency TCP 7000
email TCP 8080
adservice recommendation payment email payment TCP 50051
product catalog TCP 3550
shipping TCP 50051
frontend adservice TCP 9555
cart TCP 7070
product catalog shipping currency
checkout TCP 5050
currency TCP 7000
product catalog TCP 3550
recommendation TCP 8080
cart shipping TCP 50051
outside frontend TCP 80/8080

recommendation product catalog TCP 3550

Redis cache

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
-
 Setting the scene…

Designing your Tenants…

Switching to Application Centric mode…
Working with ESGs…
Understanding ACI security…
External Connectivity…
Increasing Security…
Automation Blueprints…
Wrapping up…

-
#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
 Design Considerations…

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
-
 Design Patterns
VRFs and BDs in “common” VRFs in “common” with
with EPGs and ESGs in the BDs, EPGs and ESGs in
Everything in the “common” “user” tenant the “user” tenant
Tenant is not typically seen
common common common

common.vrf-01 common.vrf-01 common.vrf-01

BD subnet(s) BD subnet(s) BD subnet(s) BD subnet(s)

AP
Objects in the common tenant
demo
Network EPG EPG
VLAN VLAN
should have unique names,
Segments (Security isolation per (Security isolation per BD subnet(s) BD subnet(s)
Bridge Domain) Bridge Domain)
e.g. common.vrf-01
demo
AP
EPG EPG AP Network EPG EPG
VLAN VLAN VLAN VLAN
EPG EPG Segments (Security isolation per (Security isolation per
(Security isolation per (Security isolation per Network VLAN VLAN
Bridge Domain) Bridge Domain) Segments Bridge Domain) Bridge Domain)
(Security isolation per (Security isolation per
Bridge Domain) Bridge Domain)
EPG EPG
VLAN VLAN
AP EPG EPG (Security isolation per (Security isolation per
VLAN VLAN
Bridge Domain) Bridge Domain)
ESG (Security isolation per (Security isolation per
Apps
Bridge Domain) Bridge Domain)
(Optional)
Security isolation across Bridge Domains

AP
AP Apps ESG
ESG (Optional)
Apps Security isolation across Bridge Domains

Used for functions which are accessible

(Optional)
Security isolation across Bridge Domains

from any Tenant

Typically, fewer larger subnets which can Dedicated subnets for tenants with VRFs
be (optionally) shared across Tenants that can be (optionally) shared by different
Tenants

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
-
 Design Patterns
Network team controls Large subnets can be
inbound/outbound routing shared across Tenants
All networking constructs
contained within a Tenant
demo shared-services shared-services

vrf-01 vrf-01 vrf-01

BD subnet(s) BD subnet(s)

AP
demo common
Network EPG EPG
VLAN VLAN
Segments (Security isolation per (Security isolation per vrf-01 common.vrf-01
Bridge Domain) Bridge Domain)
BD subnet(s) BD subnet(s) BD subnet(s)

EPG EPG
VLAN VLAN AP
(Security isolation per (Security isolation per
Bridge Domain) Bridge Domain) Network EPG EPG
VLAN VLAN
Segments (Security isolation per (Security isolation per
Bridge Domain) Bridge Domain)
demo test
AP
EPG EPG AP AP
VLAN VLAN
Apps ESG (Security isolation per (Security isolation per Network EPG Network EPG
(Optional) Bridge Domain) Bridge Domain) VLAN VLAN
Security isolation across Bridge Domains Segments Segments
(Security isolation per (Security isolation per
Bridge Domain) Bridge Domain)

AP EPG EPG
VLAN VLAN
Apps ESG (Security isolation per (Security isolation per
Dedicated VRFs and subnets for each (Optional)
Security isolation across Bridge Domains
Bridge Domain) Bridge Domain)

Tenant with Dedicated L3outs

AP AP
Each Tenant has one or more Apps ESG Apps ESG
network security groups Dedicated VRFs and subnets for each (Optional) Security isolation (Optional) Security isolation
across Bridge Domains across Bridge Domains
Tenant with Shared L3out

EPG and ESG in the “user” Tenant with the

VRF in the “common” Tenant, and a Shared
Each Tenant has one or more
L3out in shared-services
endpoint security groups

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
-
 Each Tenant has their own IP Range

IP range per Tenant

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
-
 Network engineers “view” of their ACI environment…

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
-
 Workloads identified by IP and Mac address
demo

vrf-01

BD 192.168.150.0_24 BD 192.168.151.0_24 BD 192.168.152.0_24 BD 192.168.153.0_24 BD 192.168.154.0_24 BD 192.168.155.0_24 BD 192.168.156.0_24

AP
EPG 192.168.150.0_24 EPG 192.168.151.0_24 EPG 192.168.152.0_24 EPG 192.168.153.0_24 EPG 192.168.154.0_24 EPG 192.168.155.0_24 EPG 192.168.156.0_24
network-segments

The “network-segments” Typical “Network Centric” mode

Application Profile contains all deployment where there is a 1:1
the EPGs which provide the mapping between Bridge
network backing Domains and EPGs

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
-
 What does the application owner care about…?

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
-
 DNS names, IP addresses, Default Gateways, and
Security Rules…

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
-
 Online Boutique
https://github.com/GoogleCloudPlatform/microservices-demo

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
-
 Online Boutique
https://github.com/GoogleCloudPlatform/microservices-demo

Source/Consumer Target/Provider Target/Provider

frontend checkout Port
cart Redis cache TCP 6379

checkout cart TCP 7070

currency TCP 7000
email TCP 8080
adservice recommendation payment email payment TCP 50051
product catalog TCP 3550
shipping TCP 50051
frontend adservice TCP 9555
cart TCP 7070
product catalog shipping currency
checkout TCP 5050
currency TCP 7000
product catalog TCP 3550
recommendation TCP 8080
cart shipping TCP 50051
outside frontend TCP 80/8080

recommendation product catalog TCP 3550

Redis cache

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
-
 Where is our application running…?
demo

vrf-01

BD 192.168.150.0_24 BD 192.168.151.0_24 BD 192.168.152.0_24 BD 192.168.153.0_24 BD 192.168.154.0_24 BD 192.168.155.0_24 BD 192.168.156.0_24

AP
EPG 192.168.150.0_24 EPG 192.168.151.0_24 EPG 192.168.152.0_24 EPG 192.168.153.0_24 EPG 192.168.154.0_24 EPG 192.168.155.0_24 EPG 192.168.156.0_24
network-segments

The application endpoints require

communication across different
subnets, which is typically achieved
frontend
using “vzAny” or “Preferred Groups”

checkout

adservice recommendation shipping email

payment

product catalog cart currency

Redis cache

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
-
 Let’s convert to “Application Centric” mode…
demo

vrf-01

BD 192.168.150.0_24 BD 192.168.151.0_24 BD 192.168.152.0_24 BD 192.168.153.0_24 BD 192.168.154.0_24 BD 192.168.155.0_24 BD 192.168.156.0_24

AP
EPG 192.168.150.0_24 EPG 192.168.151.0_24 EPG 192.168.152.0_24 EPG 192.168.153.0_24 EPG 192.168.154.0_24 EPG 192.168.155.0_24 EPG 192.168.156.0_24
network-segments

The application endpoints communicate

AP openly within the Endpoint Security Group
even though they’re connected to different
ESG all-services

frontend
Bridge Domains

checkout

adservice recommendation shipping email

online-boutique

payment

product catalog cart currency

New Application Profile created

for the application Endpoint
Security Group
Redis cache

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
-
 What does this mean to the network admin…?

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
-
 Application Visibility…!

Application Endpoint Host Information VLAN Information Tag Information

IP and MAC Information

Endpoint names EPG/Subnet Information

Switch and Interface

Information

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
-
 Correlate Endpoints to Switch Interfaces…

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
-
 What if don’t want my classification to be this granular…?

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
-
 Broad-brush classification…
demo

vrf-01

BD 192.168.150.0_24 BD 192.168.151.0_24 BD 192.168.152.0_24 BD 192.168.153.0_24 BD 192.168.154.0_24 BD 192.168.155.0_24 BD 192.168.156.0_24

AP
EPG 192.168.150.0_24 EPG 192.168.151.0_24 EPG 192.168.152.0_24 EPG 192.168.153.0_24 EPG 192.168.154.0_24 EPG 192.168.155.0_24 EPG 192.168.156.0_24
network-segments

AP
ESG production-workloads

Production workloads
production

AP
ESG pre-production-workloads

Pre-Production
workloads
pre-production

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 55
-
 What if I’d like to gather data on a specific group of
endpoints…?

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 56
-
 Endpoint classification for monitoring…
demo

vrf-01

BD 192.168.150.0_24 BD 192.168.151.0_24 BD 192.168.152.0_24 BD 192.168.153.0_24 BD 192.168.154.0_24 BD 192.168.155.0_24 BD 192.168.156.0_24

AP
EPG 192.168.150.0_24 EPG 192.168.151.0_24 EPG 192.168.152.0_24 EPG 192.168.153.0_24 EPG 192.168.154.0_24 EPG 192.168.155.0_24 EPG 192.168.156.0_24
network-segments

AP
ESG all-services

Production workloads
online-boutique

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
-
 Endpoint classification for monitoring…
demo

vrf-01

BD 192.168.150.0_24 BD 192.168.151.0_24 BD 192.168.152.0_24 BD 192.168.153.0_24 BD 192.168.154.0_24 BD 192.168.155.0_24 BD 192.168.156.0_24

AP
EPG 192.168.150.0_24 EPG 192.168.151.0_24 EPG 192.168.152.0_24 EPG 192.168.153.0_24 EPG 192.168.154.0_24 EPG 192.168.155.0_24 EPG 192.168.156.0_24
network-segments

AP
ESG all-services

Production workloads
online-boutique

ESG
monitor

Monitoring Group with

Intra ESG Contract

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
-
 Setting the scene…
Designing your Tenants…

Switching to Application
Centric mode…
Working with ESGs…
Understanding ACI security…
External Connectivity…
Increasing Security…
Automation Blueprints…
Wrapping up…

-
 All we need are the application names and the
associated IP addresses…!

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
-
 Application Knowledge taken from any source

Application Application Monitoring Application Security CMDB DNS

Knowledge e.g. AppDynamics e.g. Secure Workload e.g. SNOW

Application name +
endpoint IP addresses

APIC
Orchestration Tag Selectors
• Endpoint MAC
• Endpoint IP
• BD subnet
• Static endpoint
• VM name
• VM Tag
vCenter
Tags/Names
IP subnet selector
EPG selector
Application name +
VM Names or VM Tags

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 61
-
 Physical or virtual workloads, with or without VMM
Integration…!

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 62
-
 You can convert to Application Centric mode in two
simple steps…

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
-
 Step 1: Create Application Profiles and Security
Groups Contract allowing open or
restricted communication
demo
Application Profile for EPG
vrf-01
mapped Endpoint Security
Groups
AP epg-matched-esg

ESG vzAny

network-segments EPG/ESG Collection

C CCI I P C CCI P

Contracts applied to vzAny

implicitly applies to all EPGs,
BD 192.168.150.0_24 BD 192.168.151.0_24 BD 192.168.152.0_24

AP
EPG EPG EPG AP online-boutique ESGs, and extEPGs in the VRF
network-
segments

192.168.150.0_24 192.168.151.0_24 192.168.152.0_24 C CCI I P

dynamic (P,S) vlans dynamic (P,S) vlans dynamic (P,S) vlans ESG all-services

New Application Profile and

Typical “Network Centric” mode Endpoint Security Group for the
deployment where there is a 1:1 “online-boutique” application
mapping between Bridge
Domains and EPGs
VDS PG VDS PG VDS PG
VDS portgroup name VDS portgroup name VDS portgroup name
192.168.150.0_24 192.168.151.0_24 192.168.152.0_24
Dynamic PVLAN (P,S) Dynamic PVLAN (P,S) Dynamic PVLAN (P,S)

ESG
Open communication between
all subnets through the
network-segments

“network-segments” ESG

ESXi cluster with VMM integration

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
-
 Step 2: Tag Workloads to move into the new
Security Group Contract allowing open or
restricted communication
demo
Application Profile for EPG
vrf-01
mapped Endpoint Security
Groups
AP epg-matched-esg

ESG vzAny

network-segments EPG/ESG Collection

C CCI I P C CCI P

Contracts applied to vzAny

implicitly applies to all EPGs,
BD 192.168.150.0_24 BD 192.168.151.0_24 BD 192.168.152.0_24

AP
EPG EPG EPG AP online-boutique ESGs, and extEPGs in the VRF
Network-
segments

192.168.150.0_24 192.168.151.0_24 192.168.152.0_24 C CCI I P

dynamic (P,S) vlans dynamic (P,S) vlans dynamic (P,S) vlans ESG all-services

New Application Profile and

Typical “Network Centric” mode Endpoint Security Group for the
deployment where there is a 1:1 “online-boutique” application
mapping between Bridge
Domains and EPGs
VDS PG VDS PG VDS PG
VDS portgroup name VDS portgroup name VDS portgroup name
192.168.150.0_24 192.168.151.0_24 192.168.152.0_24
Dynamic PVLAN (P,S) Dynamic PVLAN (P,S) Dynamic PVLAN (P,S)

ESG
Open communication between
all subnets through the
network-segments

“network-segments” ESG

ESG
online-boutique

Open communication between

online-boutique endpoints
ESXi cluster with VMM integration
#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 65
-
 Tagging Option 1: Static Tag Mapping (manual/automated)
Define ESG Tag Selector

ApplicationName = online-boutique

ACI Application
Workload Tags Map MAC or IP address to Tag Value

00:50:56:A1:0A:90 = ApplicationName online-boutique

Match Endpoints to
Workload Tags
#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 66
-
 Automated conversion to “Application Centric”

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
-
 Tagging Option 2: VMM Tag Mapping
vCenter Application
Workload Tags

ACI Application
Workload Tags

ACI Application Workload Tags match

vCenter Application Workload Tags

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 68
-
 Automated conversion to “Application Centric”

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
-
 Scaling application connectivity with vzAny…

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
-
 Scaling connectivity to “application-01”
demo

vrf-01

BD 192.168.150.0_24 BD 192.168.151.0_24 BD 192.168.152.0_24

L3Out
AP
EPG EPG EPG extEPG
Network-
segments

192.168.150.0_24 192.168.151.0_24 192.168.152.0_24 0.0.0.0/1

dynamic (P,S) vlans dynamic (P,S) vlans dynamic (P,S) vlans 128.0.0.0/1
C CCI I P C CCI I P C CCI I P AP epg-matched-esg C CCI I P

ESG

network-segments

C CCI I P

vzAny as a contract consumer defines

All applications initially provide the same
that all EPGs, ESGs, extEPGs are
contract to vzAny. This maintains open
C CCI P

consumers of the same contract

vzAny

communication between applications All EPGs, ESGs, extEPGs

C CCI P

Cont
permit-to-all-applications

AP application-01 AP application-02 AP application-03

C CCI I P C CCI I P C CCI I P
ESG all-services ESG all-services ESG all-services

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
-
 Scaling connectivity to “application-02”
demo

vrf-01

BD 192.168.150.0_24 BD 192.168.151.0_24 BD 192.168.152.0_24

L3Out
AP
EPG EPG EPG extEPG
Network-
segments

192.168.150.0_24 192.168.151.0_24 192.168.152.0_24 0.0.0.0/1

dynamic (P,S) vlans dynamic (P,S) vlans dynamic (P,S) vlans 128.0.0.0/1
C CCI I P C CCI I P C CCI I P AP epg-matched-esg C CCI I P

ESG

network-segments

C CCI I P

vzAny as a contract consumer defines

C CCI P that all EPGs, ESGs, extEPGs are
consumers of the same contract
vzAny

All applications initially provide the same All EPGs, ESGs, extEPGs

contract to vzAny. This maintains open C CCI P

communication between applications

Cont
permit-to-all-applications

AP application-01 AP application-02 AP application-03

C CCI I P C CCI I P C CCI I P
ESG all-services ESG all-services ESG all-services

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 72
-
 Scaling connectivity to “application-03”
demo

vrf-01

BD 192.168.150.0_24 BD 192.168.151.0_24 BD 192.168.152.0_24

L3Out
AP
EPG EPG EPG extEPG
Network-
segments

192.168.150.0_24 192.168.151.0_24 192.168.152.0_24 0.0.0.0/1

dynamic (P,S) vlans dynamic (P,S) vlans dynamic (P,S) vlans 128.0.0.0/1
C CCI I P C CCI I P C CCI I P AP epg-matched-esg C CCI I P

ESG

network-segments

C CCI I P

vzAny as a contract consumer defines

C CCI P that all EPGs, ESGs, extEPGs are
consumers of the same contract
vzAny

All EPGs, ESGs, extEPGs

C CCI P

Cont

All applications initially provide the same permit-to-all-applications

contract to vzAny. This maintains open

communication between applications
AP application-01 AP application-02 AP application-03
C CCI I P C CCI I P C CCI I P
ESG all-services ESG all-services ESG all-services

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 73
-
 Setting the scene…
Designing your Tenants…
Switching to Application Centric mode…

Working with ESGs…

Understanding ACI security…
External Connectivity…
Increasing Security…
Automation Blueprints…
Wrapping up…

-
 Why are ESGs a better classification option…?

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 75
-
 EPG Security vs ESG Security
Bridge Domain with 1x Bridge Domain with 1x Bridge Domain with
ACI foundational building blocks: subnet and 1x EPG/vlan subnet and multiple multiple subnets and
EPGs/vlans multiple EPGs/vlans
• A Tenant provides an RBAC boundary typically linked to a
business function demo

• A VRF is mapped to a single Tenant vrf-01

• A Bridge Domain is mapped to a single VRF BD 192.168.1.1/24 BD 192.168.2.1/24 BD 192.168.3.1/24

• A Bridge Domain provides one or more IP gateways (IP

192.168.4.1/24 sec
AP
EPG EPG EPG
secondary) network-
segments Static Path 101/1/1 – vlan-10 Static Path 102/1/1 – vlan-20
Static Path 103/1/1 – vlan-30
Static Path 103/1/2 - vlan-40
• An EPG is mapped to a single Bridge Domain
An EPG provides network backing and maps to:
EPG EPG
• vmm domain vmm domain

• VMM domains + static or dynamic VLAN(s)

dynamic vlan allocation dynamic vlan allocation
EPG provides security
• Static path(s) + static VLAN(s) across a BD
EPG vmm domain
dynamic vlan allocation

An EPG defines a security boundary on a Bridge Domain

Static Path 104/1/1 – vlan-50
•
• An EPG allows open communication for endpoints in the EPG, or AP

(optionally) blocked communication for endpoints in the EPG

ESG
security-
groups Security isolation across Bridge Domains
• Inter EPG communication requires contracts (typically not required
when using ESGs)
• An ESG forms a security boundary on a VRF
• An ESG allows open communication for endpoints in the ESG, or ESGs provides security
across the VRF
(optionally) blocked communication for endpoints in the ESG
• Inter ESG communication requires contracts
• ESG contracts supersede EPG contracts

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 76
-
 What are our endpoint mapping options…?

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 77
-
 We can use EPGs, Tagged endpoints, Tagged
subnets, or simply Static endpoint mapping…

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 78
-
 Option 1: EPG mapping to a single security zone
demo
vrf-01
AP epg-matched-esg
All EPGs mapped to
ESG
a single ESG
all-subnets
EPG: 192.168.150.0_24
EPG: 192.168.151.0_24
EPG: 192.168.152.0_24

BD 192.168.150.0_24 BD 192.168.151.0_24 BD 192.168.152.0_24

AP network-segments EPG EPG EPG
192.168.150.0_24 192.168.151.0_24 192.168.152.0_24
VMM Domain VMM Domain VMM Domain
Dynamic PVLAN Dynamic PVLAN Dynamic PVLAN

Logical grouping by VDS PG VDS PG VDS PG

EPGs VDS portgroup name

192.168.150.0_24
VDS portgroup name
192.168.151.0_24
VDS portgroup name
192.168.152.0_24
PVLAN (P, S) PVLAN (P, S) PVLAN (P, S)
ESG all-subnets
EPG

Settings:
- VMM Domain (read/write)
- Allow uSegmentation = True
- Dynamic PVLANs
ESXi cluster with VMM integration

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 79
-
 Option 2: EPG mapping for multiple security zones
demo
vrf-01
AP epg-matched-esg

EPG group-01 ESG ESG EPG group-02

production Cont
pre-production
EPG: 192.168.150.0_24 permit-to-pre-production EPG: 192.168.152.0_24
EPG: 192.168.151.0_24

C CCI I P C CCI I P

BD 192.168.150.0_24 BD 192.168.151.0_24 BD 192.168.152.0_24

AP network-segments EPG EPG EPG
192.168.150.0_24 192.168.151.0_24 192.168.152.0_24
VMM Domain VMM Domain VMM Domain
Dynamic PVLAN Dynamic PVLAN Dynamic PVLAN

Logical grouping by VDS PG VDS PG VDS PG

EPGs VDS portgroup name VDS portgroup name VDS portgroup name
192.168.150.0_24
PVLAN (P, S)
192.168.151.0_24
PVLAN (P, S)
192.168.152.0_24
PVLAN (P, S) Logical grouping by
ESG production ESG pre-production EPGs
EPG

Settings:
- VMM Domain (read/write)
- Allow uSegmentation = True
- Dynamic PVLANs
ESXi cluster with VMM integration

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 80
-
 Option 3: Tag selectors with VMM integration
demo
vrf-01
AP applications

ESG ESG
application-01 Cont
application-02
VMs matched with tag VMs matched with tag
permit-to-application-02
Key: app Key: app
Value: application-01 Value: application-02
C CCI I P C CCI I P

BD 192.168.150.0_24 BD 192.168.151.0_24 BD 192.168.152.0_24

AP network-segments EPG EPG EPG
192.168.150.0_24 192.168.151.0_24 192.168.152.0_24
VMM Domain VMM Domain VMM Domain
Dynamic PVLAN Dynamic PVLAN Dynamic PVLAN

Logical grouping by VDS PG VDS PG VDS PG

VM Tag VDS portgroup name

192.168.150.0_24
VDS portgroup name
192.168.151.0_24
VDS portgroup name
192.168.152.0_24
PVLAN (P, S) PVLAN (P, S) PVLAN (P, S)
ESG application-01 VM vCenter Tag = APIC Policy Tag
EPG Key: app
Value: application-01
Settings:
ESG application-02 VM vCenter Tag = APIC Policy Tag
- VMM Domain (read/write) Key: app
- Allow uSegmentation = True Value: application-02
- Dynamic PVLANs
ESXi cluster with VMM integration

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 81
-
 Option 4: Tag selectors with VMM integration and
Intermediary switches
demo

vrf-01
AP applications

ESG ESG
application-01 Cont
application-02
VMs matched with tag VMs matched with tag
permit-to-application-02
Key: app Key: app
Value: application-01 Value: application-02
C CCI I P C CCI I P

BD BD BD
Static PVLAN
192.168.150.0_24 192.168.151.0_24 192.168.152.0_24
AP network-segments EPG EPG EPG
192.168.150.0_24 192.168.151.0_24 192.168.152.0_24
VMM Domain VMM Domain VMM Domain
Static PVLAN Static PVLAN Static PVLAN

Static PVLAN on
intermediary switches

Logical grouping by VDS PG VDS PG VDS PG

VM Tag VDS portgroup name

192.168.150.0_24
VDS portgroup name
192.168.151.0_24
VDS portgroup name
192.168.152.0_24
PVLAN (P, S) PVLAN (P, S) PVLAN (P, S)
ESG application-01 VM vCenter Tag = APIC Policy Tag
EPG Key: app
Value: application-01
Settings:
ESG application-02 VM vCenter Tag = APIC Policy Tag
- VMM Domain (read/write) Key: app
- Allow uSegmentation = True Value: application-02
- Manual/static PVLANs
ESXi cluster with VMM integration

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 82
-
 Option 5: MAC selectors, no VMM integration
demo
vrf-01
AP applications

ESG ESG
application-01 Cont
application-02
Tag Selector Tag Selector
permit-to-application-02
Key: app Key: app
Value: application-01 Value: application-02
C CCI I P C CCI I P

APIC Policy Tags:

• app:application1 -> MAC A, B, C, …
• app:application2 -> MAC X, Y, Z, … BD 192.168.150.0_24 BD 192.168.151.0_24 BD 192.168.152.0_24
AP network-segments EPG EPG EPG
192.168.150.0_24 192.168.151.0_24 192.168.152.0_24
Phys Domain Phys Domain Phys Domain
Assign an APIC policy tag to Manual PVLAN Manual PVLAN Manual PVLAN

each MAC statically on APIC

Logical grouping by VDS PG VDS PG VDS PG

MAC Tag VDS portgroup name

192.168.150.0_24
VDS portgroup name
192.168.151.0_24
VDS portgroup name
192.168.152.0_24
PVLAN (P, S) PVLAN (P, S) PVLAN (P, S)
ESG application-01 VMs MACs matched with tag
EPG Key: app
Value: application-01
Settings:
ESG application-02 VMs MACs matched with tag
- Physical Domain Key: app
- Static path bindings Value: application-02
- Manual/static PVLANs
- Intra EPG Isolation = True ESXi cluster without VMM integration
- Proxy ARP = True

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 83
-
 Option 6: IP selectors, no VMM integration
demo
vrf-01
AP applications

ESG ESG
application-01 Cont
application-02
Tag Selector Tag Selector
permit-to-application-02
Key: app Key: app
Value: application-01 Value: application-02
C CCI I P C CCI I P

APIC Policy Tags:

• app:application1 -> IP A, B, C, …
• app:application2 -> IP X, Y, Z, … BD 192.168.150.0_24 BD 192.168.151.0_24 BD 192.168.152.0_24 Intra EPG Isolation = True
AP network-segments EPG EPG EPG Proxy ARP = True
192.168.150.0_24 192.168.151.0_24 192.168.152.0_24
Phys Domain Phys Domain Phys Domain
Assign an APIC policy tag to Manual PVLAN Manual PVLAN Manual PVLAN

each IP statically on APIC

Logical grouping by VDS PG VDS PG VDS PG

IP Tag VDS portgroup name

192.168.150.0_24
VDS portgroup name
192.168.151.0_24
VDS portgroup name
192.168.152.0_24
PVLAN (P, S) PVLAN (P, S) PVLAN (P, S)
ESG application-01 VMs IPs matched with tag
EPG Key: app
Value: application-01
Settings:
ESG application-02 VMs IPs matched with tag
- Physical Domain Key: app
- Static path bindings Value: application-02
- Manual/static PVLANs
- Intra EPG Isolation = True ESXi cluster without VMM integration
- Proxy ARP = True

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 84
-
 Option 7: MAC selectors for bare metal
demo
vrf-01
AP applications

ESG ESG
application-01 Cont
application-02
Tag Selector Tag Selector
permit-to-application-02
Key: app Key: app
Value: application-01 Value: application-02
C CCI I P C CCI I P

APIC Policy Tags:

• app:application1 -> MAC A, B, C, …
• app:application2 -> MAC X, Y, Z, … BD 192.168.150.0_24 BD 192.168.151.0_24 BD 192.168.152.0_24
AP network-segments EPG EPG EPG

192.168.150.0_24 192.168.151.0_24 192.168.152.0_24

Phys Domain Phys Domain Phys Domain
Assign an APIC policy tag to
each MAC statically on APIC

ESG application-01 VLAN VLAN VLAN Bare metal MACs matched to the APIC tag
Key: app
Value: application-01

EPG ESG application-02 Bare metal MACs matched to the APIC tag
Key: app
Settings: Value: application-02

- Physical Domain Logical grouping by

- Static path bindings MAC Tag

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 85
-
 Option 8: IP selectors for bare metal
demo
vrf-01
AP applications

ESG ESG
application-01 Cont
application-02
Tag Selector Tag Selector
permit-to-application-02
Key: app Key: app
Value: application-01 Value: application-02
C CCI I P C CCI I P

APIC Policy Tags:

• app:application1 -> IP A, B, C, …
• app:application2 -> IP X, Y, Z, … BD 192.168.150.0_24 BD 192.168.151.0_24 BD 192.168.152.0_24 Intra EPG Isolation = True
AP network-segments EPG EPG EPG Proxy ARP = True
192.168.150.0_24 192.168.151.0_24 192.168.152.0_24
Phys Domain Phys Domain Phys Domain
Assign an APIC policy tag to
each IP statically on APIC

ESG application-01 VLAN VLAN VLAN Bare metal IPs matched to the APIC tag
Key: app
Value: application-01

EPG ESG application-02 Bare Metal IPs matched to the APIC tag
Key: app
Settings: Value: application-02

- Physical Domain Logical grouping by

- Static path bindings IP Tag
- Intra EPG Isolation = True
- Proxy ARP = True

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 86
-
 Option 9: Subnet selectors with mixed Domains
demo
vrf-01
AP applications
AP default-zone
ESG ESG
ESG application-01 Cont
application-02
default-zone 192.168.150.128/26 192.168.150.192/26
permit-to-application-02
192.168.151.128/26 192.168.151.192/26
192.168.150.0/25
192.168.152.128/26 192.168.152.192/26
192.168.151.0/25
192.168.152.0/25 C CCI I P C CCI I P
C CCI I P

Cont
BD 192.168.150.0_24 BD 192.168.151.0_24 BD 192.168.152.0_24
permit-to-application-01
AP network-segments EPG EPG EPG
192.168.150.0_24 192.168.151.0_24 192.168.152.0_24
VMM Domain VMM Domain Phys Domain
Dynamic PVLAN Manual PVLAN Manual PVLAN

VDS PG VDS PG Static PVLAN on

VDS portgroup name
192.168.150.0_24
VDS portgroup name
192.168.151.0_24
intermediary switches
PVLAN (P, S) PVLAN (P, S)
ESG application-01 VLAN Endpoints matched to subnets
EPG Key: app EPG
Value: application-01
Settings: ESG
Settings:
application-02 Endpoints matched to subnets
- VMM Domain Key: app
- Physical Domain
- Allow uSegmentation = True Value: application-02 - Static path bindings
- Manual/static PVLANs ESG
- Manual/static PVLANs
default-zone Endpoints matched to subnets
Key: default - Intra EPG Isolation = True
Value: default-zone - Proxy ARP = True

ESXi cluster with VMM integration

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 87
-
 Option 10: Combined solution with/without VMM
demo
vrf-01
EPG matched default AP applications
security zone AP default-zone
ESG ESG
ESG application-01 Cont
application-02
default-zone VMs and BM Tag VMs and BM Tag
permit-to-application-02
Key: app Key: app
Value: application-01 Value: application-02
C CCI I P
C CCI I P C CCI I P

APIC Policy Tags:

• app:application1 -> IP A, B, C, …
Cont
BD 192.168.150.0_24 BD 192.168.151.0_24 BD 192.168.152.0_24 • app:application2 -> IP X, Y, Z, …
permit-to-application-01
AP network-segments EPG EPG EPG
192.168.150.0_24 192.168.151.0_24 192.168.152.0_24
VMM Domain VMM Domain Phys Domain
Dynamic PVLAN Manual PVLAN Manual PVLAN
Assign an APIC policy tag to
each IP statically on APIC

VDS PG VDS PG Static PVLAN on

VDS portgroup name
192.168.150.0_24
VDS portgroup name
192.168.151.0_24
intermediary switches
PVLAN (P, S) PVLAN (P, S)
ESG application-01 VLAN VMs and IPs matched with tag
EPG Key: app EPG
Value: application-01
Settings: ESG
Settings:
application-02 VMs and IPs matched with tag
- VMM Domain Key: app
- Physical Domain
- Allow uSegmentation = True Value: application-02 - Static path bindings
- Manual/static PVLANs ESG
- Manual/static PVLANs
default-zone
- Intra EPG Isolation = True
- Proxy ARP = True

ESXi cluster with VMM integration

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 88
-
 Option 11: Combined solution + Quarantine
demo Isolated ESG to
vrf-01 prevent E/W traffic
EPG matched default AP applications
security zone AP default-zone
ESG ESG
AP quarantine

ESG application-01 Cont

application-02 ESG

default-zone VMs and BM Tag VMs and BM Tag quarantine

permit-to-application-02
Key: app Key: app
Value: application-01 Value: application-02
C CCI I P
C CCI I P C CCI I P

Cont
Assign an APIC policy tag to
BD BD BD
quarantine endpoints – match based
192.168.150.0_24 192.168.151.0_24 192.168.152.0_24
permit-to-application-01
AP
on VM Tag, VM name, MAC, IP
network-segments EPG EPG EPG
192.168.150.0_24 192.168.151.0_24 192.168.152.0_24
VMM Domain VMM Domain Phys Domain
Dynamic PVLAN Manual PVLAN Manual PVLAN

VDS PG VDS PG
VDS portgroup name VDS portgroup name Static PVLAN on
192.168.150.0_24
PVLAN (P, S)
192.168.151.0_24
PVLAN (P, S) intermediary switches
ESG application-01 VLAN

VMs and IPs matched with tag

EPG Key: app EPG
ESG application-02 Value: application-01
Settings: Settings:
VMs and IPs matched with tag
- VMM Domain Key: app
- Physical Domain
- Allow uSegmentation = True ESG default-zone Value: application-02 - Static path bindings
- Manual/static PVLANs - Manual/static PVLANs
ESG
- Intra EPG Isolation = True
quarantine
- Proxy ARP = True
VMs and IPs matched with tag
Key: endpoint
ESXi cluster with VMM integration Value: quarantine

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 89
-
 Why do we need to enable Proxy ARP for IP
mapping…

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 90
-
 MAC addresses are not classified to ESGs when only IP-based
selectors are used. Switching traffic (i.e. within the same subnet)
will not use ESG contracts even if its payload has the IP address
classified to an ESG…

If two IPs in the same subnet from the same EPG are classified
into different ESGs, those two endpoints can still talk freely
through the MAC and its original EPG…

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 91
-
 Proxy ARP
demo
vrf-01
AP epg-matched-esg

ESG ESG

IP based ESG IP based ESG

192.168.150.21 192.168.150.22

BD 192.168.150.0_24
AP network-segments
EPG

192.168.150.21 192.168.150.22

00:00:00:00:00:21 00:00:00:00:00:22

MAC to MAC
allowed

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 92
-
 How do you enable Proxy ARP on the Leaf
Switches…?
Enabling “Allow Micro-Segmentation”
automatically enables Proxy ARP.
Option in a 100% virtual deployment, use
with or without Intra EPG isolation

Enable Intra EPG isolation with

Proxy ARP if you have a mixed
virtual and physical environment Enabling Intra EPG isolation / Allow Micro-
Segmentation configures PVLANs on the
port group

Proxy ARP is only available Add an Intra EPG

when Intra ESG isolation is Contract
enabled

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 93
-
 vCenter tag/name matching requires read/write
vmm integration…

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 95
-
 Dynamic Policy Tag matching from vCenter

Create ACI Tags to

match vCenter Tags

Tag Collection runs

every 5 min

Tenant → Policies → Endpoint Tags

APIC creates dynamic VMM MAC Tags based

on the assigned Category / Tag in vCenter #CiscoLiveAPJC © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 96
-
BRKDCN-2984
 Static endpoint mapping…

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 98
-
 Static Policy Tags on APIC

Static Endpoints

IP address ranges

MAC addresses

IP addresses

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 99
-
 What if you have a Greenfield deployment…?

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 103
-
 Greenfield option – 1:1 EPG to ESG
mapping
demo
vrf-01
AP network-security-groups

ESG ESG ESG

Contracts on
Network security ESGs
groups 192.168.150.0_24 192.168.151.0_24 192.168.152.0_24

Network backing BD 192.168.150.0_24 BD 192.168.151.0_24 BD 192.168.152.0_24

(VLANs) AP
No contracts on
network-segments
EPGs
EPG EPG EPG
192.168.150.0_24 192.168.151.0_24 192.168.152.0_24
VMM Domain VMM Domain VMM Domain
Dynamic PVLAN Dynamic PVLAN Dynamic PVLAN

VDS PG VDS PG VDS PG

VDS portgroup name VDS portgroup name VDS portgroup name
192.168.150.0_24 192.168.151.0_24 192.168.152.0_24
PVLAN (P, S) PVLAN (P, S) PVLAN (P, S)

EPG

Settings:
- VMM Domain (read/write)
- Allow uSegmentation = True
- Dynamic PVLANs
ESXi cluster with VMM integration

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 104
-
 Consider automated static MAC tagging derived
from the endpoint IP address…

Works for Bare Metal and VMs, with or without

VMM Integration…

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 105
-
 Understanding ACI
security…

-
 Allowing open
communication…

-
 There are four options to allow open
communication…
https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-743951.html#Migrationexample

• vzAny
• Preferred Groups
• EPGs mapped Endpoint Security Groups
• Disable security (not covered, because why would you…?)

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 117
-
 Existing applications typically require unrestricted
communication
demo

vrf-01

BD 192.168.150.0_24 BD 192.168.151.0_24 BD 192.168.152.0_24 BD 192.168.153.0_24 BD 192.168.154.0_24 BD 192.168.155.0_24 BD 192.168.156.0_24

AP
EPG 192.168.150.0_24 EPG 192.168.151.0_24 EPG 192.168.152.0_24 EPG 192.168.153.0_24 EPG 192.168.154.0_24 EPG 192.168.155.0_24 EPG 192.168.156.0_24
network-segments

The application endpoints require

communication across different
subnets, which is typically achieved
frontend
using “vzAny” or “Preferred Groups”

checkout

adservice recommendation shipping email

payment

product catalog cart currency

Redis cache

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 118
-
 vzAny

BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 119
-
 The great thing about vzAny provide/consume is that it
allows open communication between all endpoints…

The “bad” thing about vzAny provide/consume is that it

allows open communication between all endpoints…!

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 120
-
 vzAny Operation – Consumer and Provider
https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/kb/b_KB_Use_vzAny_to_AutomaticallyApplyCommunicationRules_toEPGs.html

common

common.vrf-01

Cont Subj Filt

common:default default default

Default contract in the “common”

tenant allows all traffic Entry
unspecified

demo

vrf-01

C CCI P
vzAny as a contract Provider and EPG
Consumer means that all EPGs (inc vzAny
extEPG) are implicitly Providers and
C CCI P
Consumers of the contract

C CCI I P C CCI I P C CCI I P

extEPG EPG ESG

All extEPGs All EPGs All ESGs

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 121
-
 vzAny Operation – Consumer and Provider
https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/kb/b_KB_Use_vzAny_to_AutomaticallyApplyCommunicationRules_toEPGs.html

common

common.vrf-01

Cont Subj Filt

common:default default default

Default contract in the “common”

tenant allows all traffic Entry
unspecified

demo

vrf-01 vrf-02

C CCI P
EPG vzAny as a contract Provider and
L3Out vzAny Consumer means that all EPGs (inc
extEPG) are implicitly Providers and
extEPG C CCI P
Consumers of the contract
0.0.0.0/1
128.0.0.0/1

C CCI I P
C CCI I P C CCI I P
EPG ESG

All EPGs All ESGs

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 122
-
 vzAny Operation – Consumer and Provider
https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/kb/b_KB_Use_vzAny_to_AutomaticallyApplyCommunicationRules_toEPGs.html

shared-services common

vrf-01 common.vrf-01 Default contract in the “common”

tenant allows all traffic

L3Out Cont Subj Filt

common:default default default
extEPG
0.0.0.0/1
128.0.0.0/1

C CCI I P Entry
unspecified

demo

vrf-01

C CCI P
vzAny as a contract Provider and EPG
Consumer means that all EPGs (inc vzAny
extEPG) are implicitly Providers and
C CCI P
Consumers of the contract

C CCI I P C CCI I P
EPG ESG

All EPGs All ESGs

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 123
-
 vzAny cannot be a Provider for Shared Services
shared-services

vrf-01

ESG

core-services
Requirement is to permit ssh from
C CCI I P “core-services” to all endpoints in any
given tenant i.e. tcp-src-any-dst-22
Cont

Name: permit-from-core-services
Subject: tcp
Filter: tcp-src-any-dst-22
Exported: Yes

Tenant-01 Tenant-02 Tenant-03

vrf-01 vrf-01 vrf-01

Cont Cont Cont

Name: permit-from-core-services Name: permit-from-core-services Name: permit-from-core-services

Subject: tcp Subject: tcp Subject: tcp
Filter: tcp-src-any-dst-22 Filter: tcp-src-any-dst-22 Filter: tcp-src-any-dst-22
Imported: Yes Imported: Yes Imported: Yes

C CCI I P C CCI I P C CCI I P

EPG EPG EPG
vzAny vzAny vzAny

Use vzAny to allow Use vzAny to allow Use vzAny to allow

SSH to all EPGs/ESGs SSH to all EPGs/ESGs SSH to all EPGs/ESGs

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 124
-
 vzAny cannot be a Provider for Shared Services
shared-services

vrf-01

ESG

core-services
Requirement is to permit ssh from
C CCI I P “core-services” to all endpoints in any
given tenant i.e. tcp-src-any-dst-22
Cont

Name: permit-from-core-services
Subject: tcp
Filter: tcp-src-any-dst-22
Exported: Yes

vzAny cannot be a provider for

Shared Services
Tenant-01 Tenant-02 Tenant-03

vrf-01 vrf-01 vrf-01

Cont Cont Cont

Name: permit-from-core-services Name: permit-from-core-services Name: permit-from-core-services

Subject: tcp Subject: tcp Subject: tcp
Filter: tcp-src-any-dst-22 Filter: tcp-src-any-dst-22 Filter: tcp-src-any-dst-22
Imported: Yes Imported: Yes Imported: Yes

C CCI I P C CCI I P C CCI I P

EPG EPG EPG
vzAny vzAny vzAny

Use vzAny to allow Use vzAny to allow Use vzAny to allow

SSH to all EPGs/ESGs SSH to all EPGs/ESGs SSH to all EPGs/ESGs

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 125
-
 vzAny can absolutely be your friend, but remember
that vzAny contract relationships are applied to all
EPGs, ESGs, extEPGs in the VRF…

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 126
-
 Preferred Groups

BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 127
-
 Preferred Groups
https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/kb/b_KB_Use_vzAny_to_AutomaticallyApplyCommunicationRules_toEPGs.html

demo

vrf-01 There is only one

preferred group per VRF

Preferred Group
BD 192.168.150.0_24 BD 192.168.151.0_24 BD 192.168.152.0_24 Enable Preferred Group
EPG EPG EPG
on VRF
192.168.150.0_24 192.168.151.0_24 192.168.152.0_24
Intra EPG = Unenforced Intra EPG = Unenforced Intra EPG = Unenforced

Typical “Network Centric” mode

deployment where there is a 1:1
mapping between Bridge
Domains and EPGs

pcTag 49160 pcTag 49159 pcTag 16393

Include EPG in Include EPG in Include EPG in

Preferred Group Preferred Group Preferred Group

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 128
-
 There can only be one “Preferred Group” per VRF...

It is not possible to add Contract relationships to a

“Preferred Group”

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 129
-
 All EPGs mapped
to a single ESG

BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 130
-
 Initial state: Isolated groups of workloads
demo

vrf-01

BD 192.168.150.0_24 BD 192.168.151.0_24 BD 192.168.152.0_24

AP
EPG EPG EPG
Network-
segments

192.168.150.0_24 192.168.151.0_24 192.168.152.0_24

dynamic vlan dynamic vlan dynamic vlan

Typical “Network Centric” mode

deployment where there is a 1:1
mapping between Bridge
Domains and EPGs
VDS PG VDS PG VDS PG
VDS portgroup name
192.168.150.0_24
VDS portgroup name
192.168.151.0_24
VDS portgroup name
192.168.152.0_24
No communication between
Dynamic VLAN Dynamic VLAN Dynamic VLAN portgroup/subnets 192.168.150.0,
192.168.151.0 and 192.168.152.0

ESXi cluster with VMM integration

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 132
-
 Enable Endpoint Security Groups Primary/Port Encap VLANs not
required for directly attached hosts

Static Primary / Encap VLANs are required

when there is an intermediary switching layer
such as UCS FIs

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 133
-
 PVLAN and MAC Tagging

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 134
-
 Each EPG has a unique security Tag (pcTag)

pcTag: 32771 pcTag: 49155 pcTag: 16390

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 135
-
 Enabling ESG micro segmentation on a read/write VMM Domain enables
PVLANs in the hypervisor to control East/West traffic…

plus, micro segmentation also enables Proxy ARP and dynamic endpoint
MAC Tagging…

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 136
-
 What’s the impact to traffic when we enable
uSegmentation…?
demo

vrf-01

EPG
vzAny
(allowing ICMP)

C CCI P

BD 192.168.150.0_24 BD 192.168.151.0_24 BD 192.168.152.0_24

Cont
permit-icmp AP
EPG EPG EPG
Network-
segments

192.168.150.0_24 192.168.151.0_24 192.168.152.0_24

dynamic (P,S) vlans dynamic (P,S) vlans dynamic (P,S) vlans
C CCI I P C CCI I P C CCI I P
vzAny allows
ICMP

VDS PG VDS PG VDS PG

VDS portgroup name VDS portgroup name Increase in latency from 0.21ms to
VDS portgroup name
192.168.150.0_24
Dynamic PVLAN (P,S)
192.168.151.0_24
Dynamic PVLAN (P,S) 814ms whilst the change takes
192.168.152.0_24
Dynamic PVLAN (P,S)
place – but zero packets are
pinger dropped

ubuntu-03
ESXi cluster with VMM integration 192.168.152.21

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 137
-
 Let’s map our EPGs to an ESG…

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 138
-
 Create an Application Profile for Security Groups

New Application Profile

for Security Groups

epg-matched-security-groups

Do not create EPGs

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 139
-
 Create a new ESG for Network Segments (EPGs)
Enter ESG name
“group-01”

Create new
ESG
Add EPGs

Select the VRF for the ESG

to be applied against

Allow Intra ESG traffic i.e.

Select one or more permit traffic between EPGs
EPGs

Finish

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 140
-
 Open communication within the ESG…
Static EPG to
demo ESG mapping
vrf-01 AP epg-matched-esg

EPG ESG
vzAny
(allowing ICMP) network-segments

C CCI P C CCI I P Single security

BD 192.168.150.0_24 BD 192.168.151.0_24 BD 192.168.152.0_24
zone
AP
EPG EPG EPG
Network-
segments

192.168.150.0_24 192.168.151.0_24 192.168.152.0_24

dynamic (P,S) vlans dynamic (P,S) vlans dynamic (P,S) vlans
C CCI I P C CCI I P C CCI I P

VDS PG VDS PG VDS PG

VDS portgroup name VDS portgroup name VDS portgroup name
192.168.150.0_24 192.168.151.0_24 192.168.152.0_24
Open communication between Dynamic PVLAN (P,S) Dynamic PVLAN (P,S) Dynamic PVLAN (P,S)

subnets 192.168.150.0 and ESG

network-segments

192.168.151.0

No communication (other than

vzAny) to subnet 192.168.152.0

ESXi cluster with VMM integration

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 141
-
 Matched EPGs now classified with a common pcTag

pcTag: 31 pcTag: 31 pcTag: 49157

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 142
-
 Let’s consider any impact to traffic when adding the
remaining EPG to the Security Group…

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 143
-
 Add remaining EPG to Single Security Zone
demo

vrf-01 AP epg-matched-esg
Add remaining EPG to ESG
the “network-segments”
EPG
vzAny
network-segments
security zone
(allowing ICMP)

C CCI P C CCI I P Single security

BD 192.168.150.0_24 BD 192.168.151.0_24 BD 192.168.152.0_24
zone
Cont
permit-icmp AP
EPG EPG EPG
Network-
segments

192.168.150.0_24 192.168.151.0_24 192.168.152.0_24

dynamic (P,S) vlans dynamic (P,S) vlans dynamic (P,S) vlans
Intra EPG = Unenforced Intra EPG = Unenforced Intra EPG = Unenforced

ICMP only permitted from ubuntu-01 to ubuntu-

C CCI I P C CCI I P C CCI I P
vzAny allows
ICMP 03 after EPG 192.168.152.0_24 is added to the
“network-segments” security zone

VDS PG VDS PG VDS PG

VDS portgroup name VDS portgroup name VDS portgroup name
192.168.150.0_24 192.168.151.0_24 192.168.152.0_24
Dynamic PVLAN (P,S) Dynamic PVLAN (P,S) Dynamic PVLAN (P,S)

ESG
network-segments
pinger

ubuntu-01 ubuntu-03
192.168.150.21 192.168.152.21
ESXi cluster with VMM integration
#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 144
-
 All EPGs now classified with a common pcTag

pcTag: 31 pcTag: 31 pcTag: 31

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 145
-
 Benefits of EPG to ESG Mapping

• More flexible than using vzAny as it is applied to specific EPGs to create one or more
security groups based on subnets/vlans*
• More secure than vzAny as EPG/ESG mapping does not include the extEPG – a contract is
required for external communication
• More integrated than vzAny as supports provider function for Shared Services
• More flexible than Preferred Groups as you can multiple ESG groups vs a single preferred
group
• More integrated than Preferred Groups as you can create a contract to the whole ESG

* Assumes 1:1 mapping between Bridge Domain and EPG. ESGs can mapping can also be performed on IP subnets

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 149
-
 Allowing restricted
communication…

-
 Let’s check our understanding on how contracts
work…

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 151
-
 How do contracts work…?

Outside Inside

ubuntu-01 ubuntu-02

permit ubuntu-01 ubuntu-02 tcp 5201

EPG Security applied
at VLAN boundary

EPG Cont EPG

vlan-10 permit-to-vlan-11 vlan-11
(pcTag 32777) (pcTag 49162)
ubuntu-01 C CCI I P C CCI I P ubuntu-02
192.168.150.21 192.168.151.21

ESG Security applied

at VRF boundary
Cont

ESG permit-to-cart-svc ESG

frontend-svc cart-svc
(pcTag 4168) (pcTag 1856)
ubuntu-01 C CCI I P C CCI I P ubuntu-02
192.168.150.21 192.168.151.21

*arrows indicate expected direction of connection i.e. from consumer to provider

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 152
-
 Consumer and Provider relationships are there to help you visualize the traffic
flow direction
i.e. (typically) from the consumer to the provider

Consumer and Provider relationships do not (by default) prevent TCP

connections being established from the Provider to the Consumer

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 153
-
 Contract Structure…
ESG ESG

frontend-svc cart-svc
ubuntu-01 C CCI I P C CCI I P ubuntu-02
192.168.150.21 192.168.151.21
Contract name typically tied to
the Provider EPG/ESG
Cont

permit-to-cart-svc

Filter/Entry name identifies

protocol, src port, and dst port
Subj Filt Entry
tcp tcp-src-any-dst-7070 tcp-src-any-dst-7070
Subject name identifies
the protocol Filt Entry
tcp-src-any-dst-443 tcp-src-any-dst-443

Subj Filt Entry

udp udp-src-any-dst-53 udp-src-any-dst-53

Subj Filt Entry

icmp icmp icmp

Subj Filt Entry

redirect tcp-src-any-dst-80 tcp-src-any-dst-80

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 154
-
 Who hasn’t simply done this…

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 156
-
 permit any/any
ESG ESG

frontend-svc cart-svc
ubuntu-01 C CCI I P C CCI I P ubuntu-02
192.168.150.21 192.168.151.21

Cont

permit-any

Subj Filt Entry

permit-any permit-any unspecified

# netcat –p [any] ubuntu-02 [any] # netcat –l [any]

ubuntu-01 ubuntu-02
192.168.150.21 192.168.151.21
# netcat –l [any] # netcat –p [any] ubuntu-01 [any]

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 157
-
 Contracts also trigger route leaking for EPGs…

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 158
-
 Contract Scope Scope = Application allows
production development
connectivity between EPGs/ESGs
vrf-01
AP
ExG
ESG
within the same Application
Cont

my-app-01
application
endpoints
Scope
App permit-any
ExG
ESG
application
endpoints
Scope = VRF allows connectivity
between EPGs/ESGs within the
vrf-02 AP AP same VRF
ExG
ESG ExG
ESG
Cont

my-app-03
my-app-02
application application
endpoints endpoints
Scope
VRF permit-any
ExG
ESG ExG
ESG
application application
endpoints endpoints
Scope = Tenant allows connectivity between
EPGs/ESGs within the same Tenant – note the
vrf-03 AP
ExG
AP
ExG
vrf-04
AP
ExG
contract also triggers route leaking
ESG ESG ESG
Cont

my-app-05
my-app-04

my-app-06
application application application
endpoints endpoints endpoints
Scope
Tenant permit-any ExG
ESG ExG
ESG ExG
ESG
application application application
endpoints endpoints endpoints

vrf-05 vrf-06 vrf-01

AP AP AP AP
ExG
ESG ExG
ESG ExG
ESG ExG
ESG
Cont
my-app-08
my-app-07

my-app-09

my-app-10
application application application application
endpoints endpoints endpoints endpoints
Scope
Global permit-any ExG
ESG ExG
ESG ExG
ESG ExG
ESG
application application application application
endpoints endpoints endpoints endpoints
Scope = Global allows connectivity between
ExG = Applies to either EPGs or ESGs
EPGs/ESGs within the between Tenants – note
the contract also triggers route leaking
#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 159
-
 Verifying Contract operation with netcat – Stateful = No
demo

vrf-01

Cont
ESG permit-to-ubuntu-02 ESG

ubuntu-01 ubuntu-02
ubuntu-01 C CCI I P C CCI I P ubuntu-02
192.168.150.21 Subj
tcp 192.168.151.21

Communication to and from Communication to and from

“any” port is allowed Filt
tcp-src-any-dst-7070 port “7070” is allowed

Stateful: No
Entry tcp-src-any-dst-7070
(Stateful: No)

# netcat –p [any] ubuntu-02 7070 # netcat –l 7070

# netcat –p [any] ubuntu-02 7070 # netcat –l 7071

ubuntu-01 ubuntu-02
# netcat –l 5000 # netcat –p 7070 ubuntu-01 5000
192.168.150.21 192.168.151.21
# netcat –l 5000 # netcat –p 7071 ubuntu-01 5000

Provider to Consumer connections

Incorrect Provider side port are allowed when the Provider side
port is specified as the source port
#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 160
-
 Verifying Contract Operation: EPG/ESG details
demo

vrf-01

pcTag: 38 pcTag: 5474

Cont
ESG permit-to-ubuntu-02 ESG

ubuntu-01 ubuntu-02
ubuntu-01 ubuntu-02
C CCI I P C CCI I P
192.168.150.21 192.168.151.21

aci-dev-01-apic-01# show esg ubuntu-01 detail aci-dev-01-apic-01# show esg ubuntu-02 detail
Endpoint Security Group Data: Endpoint Security Group Data:
Tenant : demo Tenant : demo
Application : endpoint-matched-security-groups Application : endpoint-matched-security-groups
ESg : ubuntu-01 ESg : ubuntu-02
VRF : vrf-01 VRF : vrf-01
Intra ESG Isolation : unenforced Intra ESG Isolation : unenforced
Policy Tag : 38 Policy Tag : 5474
Consumed Contracts : permit-to-ubuntu-02 Consumed Contracts :
Provided Contracts : Provided Contracts : permit-to-ubuntu-02
Consumed Contracts Interface : Consumed Contracts Interface :
Qos Class : unspecified Qos Class : unspecified
Tag List : Tag List :

IP Selectors: IP Selectors:
Name Match Expression Name Match Expression
-------------------- ----------------------------------------- -------------------- -----------------------------------------
ip=='192.168.150.21’ ip=='192.168.151.21’

!output truncated !output truncated

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 161
-
 Verifying Contract Operation: Contract details
demo

vrf-01

pcTag: 38 pcTag: 5474

Cont
ESG permit-to-ubuntu-02 ESG

ubuntu-01 ubuntu-02
ubuntu-01 ubuntu-02
C CCI I P C CCI I P
192.168.150.21 192.168.151.21

aci-dev-01-apic-01# show contract permit-to-ubuntu-02

Tenant Contract Type Qos Class Scope Subject Access-group Dir Description
---------- ---------- ---------- ------------ ---------- ---------- ---------- ---- ----------
demo permit-to-ubuntu-02 permit unspecified vrf icmp icmp both
demo permit-to-ubuntu-02 permit unspecified vrf tcp tcp-src-any-dst-7070 both

Subject: icmp
Subject: tcp
aci-dev-01-apic-01# show access-list tcp-src-any-dst-7070
Tenant : demo Scope: VRF
Access-List : tcp-src-any-dst-7070
match tcp dest 7070

Access Control
Entry

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 162
-
 Verifying Contract Operation: Drop details
demo

vrf-01

pcTag: 38 pcTag: 5474

Cont
ESG permit-to-ubuntu-02 ESG

ubuntu-01 ubuntu-02
ubuntu-01 ubuntu-02
C CCI I P Show ACL deny C CCI I P
192.168.150.21 192.168.151.21
log

aci-dev-01-apic-01# show acllog deny l3 flow tenant demo vrf vrf-01 srcip 192.168.150.21
SrcIp DstIp Protocol SrcPort DstPort Node SrcIntf VrfEncap
-------------- -------------- -------- ----------- ----------- ---------- ------------ -------------
192.168.150.21 129.250.35.250 udp 38849 123 101 Ethernet1/31 VXLAN:2129922

192.168.150.21 23.94.219.146 udp 48979 123 101 Ethernet1/31 VXLAN:2129922

192.168.150.21 84.245.9.254 udp 39062 123 101 Ethernet1/31 VXLAN:2129922

192.168.150.21 149.210.142.45 udp 44073 123 101 Ethernet1/31 VXLAN:2129922

192.168.150.21 164.92.216.152 udp 50220 123 101 Ethernet1/31 VXLAN:2129922

SrcIP DstIP Protocol SrcPort DstPort

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 164
-
 Verifying Contract operation with netcat – Stateful = Yes
demo

vrf-01

Cont
ESG permit-to-ubuntu-02 ESG

ubuntu-01 ubuntu-02
ubuntu-01 C CCI I P C CCI I P ubuntu-02
192.168.150.21 Subj
tcp 192.168.151.21

Communication to and from Communication to and from

“any” port is allowed Filt
tcp-src-any-dst-7070 port “7070” is allowed

Stateful: Yes
Entry tcp-src-any-dst-7070
(Stateful: Yes)

# netcat –p [any] ubuntu-02 7070 # netcat –l 7070

# netcat –p [any] ubuntu-02 7070 # netcat –l 7071

ubuntu-01 ubuntu-02
# netcat –l 5000 # netcat –p 7070 ubuntu-01 5000
192.168.150.21 192.168.151.21
# netcat –l 5000 # netcat –p 7071 ubuntu-01 5000
Provider to Consumer connections
are blocked as the contract is a
Incorrect Provider side port “stateful” contract

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 165
-
 Verifying Contracts with Syslog and ELAM

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 166
-
 Cont Cont

permit-to-ubuntu-02 permit-to-ubuntu-02

Filter Entry source port = port opened on

Subj

tcp
the consumer EPG/ESG Subj

tcp

Filter Entry destination port = port opened on

Filt

tcp-src-any-dst-7070 the provider EPG/ESG tcp-src-7070-dst-any

Filt

Entry Entry

src=any | dst=7070 src=7070 | dst=any

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 167
-
 Getting into the
weeds…!

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 168
 Reversing the Filter ports – Stateful = No
demo

vrf-01

Cont
ESG permit-to-ubuntu-02 ESG

ubuntu-01 ubuntu-02
ubuntu-01 C CCI I P C CCI I P ubuntu-02
192.168.150.21 Subj
tcp 192.168.151.21

Communication to and from Communication to and from

port “22” is allowed Filt
tcp-src-22-dst-any “any” port is allowed

Stateful: No
Source port must be Entry tcp-src-22-dst-any
“22” (Stateful: No)

# netcat –p 22 ubuntu-02 5000 # netcat –l 5000

# netcat –l 22 # netcat –p [any] ubuntu-01 22

ubuntu-01 ubuntu-02
# netcat –l 5000 # netcat –p [any] ubuntu-01 5000
192.168.150.21 192.168.151.21

Incorrect Consumer side port

Provider to Consumer connections
are allowed when the Provider side
port is specified as the source port
#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 169
-
 Why would you want to reverse the Consumer
and Provider Filters…?

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 170
-
 vzAny as a contract Provider
src_port = port open on the consumer EPG/ESG shared-services
dsr_port = port open on the provider side EPG/ESG vrf-01

ESG

core-services
Requirement is to permit ssh from
C CCI I P “core-services” to all endpoints in any
given tenant i.e. tcp-src-any-dst-22
Cont

Name: permit-from-core-services
Subject: tcp
Filter: tcp-src-any-dst-22
Exported: Yes

Tenant-01 Tenant-02 Tenant-03

vrf-01 vrf-01 vrf-01

Cont Cont Cont

Name: permit-from-core-services Name: permit-from-core-services Name: permit-from-core-services

Subject: tcp Subject: tcp Subject: tcp
Filter: tcp-src-any-dst-22 Filter: tcp-src-any-dst-22 Filter: tcp-src-any-dst-22
Imported: Yes Imported: Yes Imported: Yes

C CCI I P C CCI I P C CCI I P

EPG EPG EPG
vzAny vzAny vzAny

Use vzAny to allow Use vzAny to allow Use vzAny to allow

SSH to all EPGs/ESGs SSH to all EPGs/ESGs SSH to all EPGs/ESGs

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 171
-
 vzAny as a contract Provider
src_port = port open on the consumer EPG/ESG shared-services
dsr_port = port open on the provider side EPG/ESG vrf-01

ESG

core-services
Requirement is to permit ssh from
C CCI I P “core-services” to all endpoints in any
given tenant i.e. tcp-src-any-dst-22
Cont

Name: permit-from-core-services
Subject: tcp
Filter: tcp-src-any-dst-22
Exported: Yes

vzAny cannot be a provider for

Shared Services
Tenant-01 Tenant-02 Tenant-03

vrf-01 vrf-01 vrf-01

Cont Cont Cont

Name: permit-from-core-services Name: permit-from-core-services Name: permit-from-core-services

Subject: tcp Subject: tcp Subject: tcp
Filter: tcp-src-any-dst-22 Filter: tcp-src-any-dst-22 Filter: tcp-src-any-dst-22
Imported: Yes Imported: Yes Imported: Yes

C CCI I P C CCI I P C CCI I P

EPG EPG EPG
vzAny vzAny vzAny

Use vzAny to allow Use vzAny to allow Use vzAny to allow

SSH to all EPGs/ESGs SSH to all EPGs/ESGs SSH to all EPGs/ESGs

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 172
-
 vzAny as a contract Consumer – Filters Reversed
src_port = port open on the consumer EPG/ESG shared-services
dsr_port = port open on the provider side EPG/ESG vrf-01

ESG

Reverse the Filter ports in core-services

Requirement is to permit ssh from
the Contract C CCI I P “core-services” to all endpoints in any
given tenant i.e. tcp-src-any-dst-22
Cont
Provide the
Name: permit-from-core-services
Subject: tcp Contract
Filter: tcp-src-22-dst-any
Exported: Yes
tcp- -dst-any
means any TCP port is open on
the Provider side
Tenant-01 tcp-src-22- Tenant-02 Tenant-03

vrf-01
means TCP port 22 is open on vrf-01 vrf-01

Cont the Consumer side Cont Cont

Name: permit-from-core-services Name: permit-from-core-services Name: permit-from-core-services

Subject: tcp Subject: tcp Subject: tcp
Filter: Consume
tcp-src-22-dst-any the Filter: tcp-src-22-dst-any Consume the Filter: tcp-src-22-dst-any
Imported: Yes
exported contract(s)
Imported: Yes Contract Imported: Yes

C CCI I P C CCI I P C CCI I P

EPG EPG EPG
vzAny vzAny vzAny

Use vzAny to allow Use vzAny to allow Use vzAny to allow

SSH to all EPGs/ESGs SSH to all EPGs/ESGs SSH to all EPGs/ESGs

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 173
-
 Setting the scene…
Designing your Tenants…
Switching to Application Centric mode…
Working with ESGs…
Understanding ACI security…

External Connectivity…
Increasing Security…
Automation Blueprints…
Wrapping up…

-
 Where should you place your L3outs…?

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 207
-
 tenant “common”, “shared-services”, or in the
“workload/user” tenant…

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 208
-
 External Connectivity

demo shared-services common

vrf-01 vrf-01 common.vrf-01

BD subnet(s) BD subnet(s) BD subnet(s) BD subnet(s)

AP AP

Network EPG EPG Network EPG EPG

VLAN VLAN VLAN VLAN
Segments (Security isolation per (Security isolation per Segments (Security isolation per (Security isolation per
demo
Bridge Domain) Bridge Domain) Bridge Domain) Bridge Domain)
vrf-01
EPG EPG EPG EPG
VLAN VLAN BD subnet(s) BD subnet(s) VLAN VLAN
(Security isolation per (Security isolation per (Security isolation per (Security isolation per
Bridge Domain) Bridge Domain) Bridge Domain) Bridge Domain)
AP

Network EPG EPG

VLAN VLAN
Segments (Security isolation per (Security isolation per
AP Bridge Domain) Bridge Domain)
Apps ESG
(Optional) demo
Security isolation across Bridge Domains EPG EPG
VLAN VLAN
(Security isolation per (Security isolation per AP
Bridge Domain) Bridge Domain)
Apps ESG
(Optional) Security isolation across Bridge Domains
(Endpoints grouped by IP address*)

Dedicated VRFs and subnets for each AP

Tenant with Dedicated L3outs Apps ESG

(Optional)
Shared networking with isolated security
Security isolation across Bridge Domains

Dedicated VRFs and subnets for each

Tenant with Shared L3out

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 209
-
 What’s in a L3out…?
External Device External Device

shared-services Switches, Router IDs, Loopback

vrf-01
addresses, Static Routes

Path 101/1/7 Path 102/1/7

10.237.99.233/30 10.237.99.237/30

L3out
vrf-01-ospf-area—0.0.0.1 Interfaces and Routing Protocols

extEPG
vrf-01-all-ext-subnets
C CCI I P

Subnet Classifier
Subnets

IP Address: 0.0.0.0/0
Cont
Scope: External Subnets for the extEPG
permit-to-online-boutique
(scope = vrf)

*arrows indicates direction of traffic flow i.e. from consumer to provider

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 211
-
 Option 1 – Dedicated L3out per Tenant

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 212
-
 Dedicated L3out
External Device External Device
Bridge Domains set to
advertise subnet
demo

vrf-01

BD 192.168.150.0_24 BD 192.168.151.0_24 BD 192.168.152.0_24

Path 101/1/7 Path 102/1/7 advertise=yes, shared=no advertise=yes, shared=no advertise=yes, shared=no
10.237.99.233/30 10.237.99.237/30
AP
EPG EPG EPG

network-
segments
192.168.150.0_24 192.168.151.0_24 192.168.152.0_24

L3out
vrf-01-ospf-area—0.0.0.1

Bridge Domains AP online-boutique

mapped to L3out ESG pcTag: 5490

extEPG pcTag: 15
IP address: Classifies remote vrf-01-all-ext-subnets all-services
endpoints/subnets C CCI I P
C CCI I P

Subnets Contract Contract

IP Address Scope Consumer Provider
0.0.0.0/0 External Subnets for the extEPG Cont
permit-to-online-boutique
(scope = vrf)
External Subnets for the extEPG:
Allows connections to/from the
endpoints/subnets through a contract

*arrows indicate expected direction of connection i.e. from consumer to provider

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 213
-
 External Classification
IP Address:
Identifies remote endpoints/subnets

External Subnets for External EPG:

Required for contract purposes

• IP Address: identifies remote endpoints/subnets

• External Subnets for External EPG: allows packets to/from the L3out with a contract

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 214
-
 Option 2 – Shared L3out

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 215
-
 Shared L3out – Route Leaking between VRFs (ESGs)
External Device External Device

shared-services demo

vrf-01 vrf-01

BD 192.168.150.0_24 BD 192.168.151.0_24 BD 192.168.152.0_24

Path 101/1/7 Path 102/1/7 advertise=yes, shared=yes advertise=yes, shared=yes advertise=yes, shared=yes
10.237.99.233/30 10.237.99.237/30
AP
EPG EPG EPG

network-
segments
192.168.150.0_24 192.168.151.0_24 192.168.152.0_24

Consumed
L3out
vrf-01-ospf-area—0.0.0.1
Contract Interface Route leak
between VRFs
AP online-boutique

extEPG pcTag: 41 Contract ESG pcTag: 5490

IP address: Classifies remote vrf-01-all-ext-subnets Exported all-services
endpoints/subnets C CCI I P
C CCI I P

Subnets Contract
IP Address Scope Provider
0.0.0.0/1 External Subnets for the extEPG Cont Cont
Shared Security Import Subnet
permit-to-tn-demo-online-boutique permit-to-tn-demo-online-boutique
128.0.0.0/1 External Subnets for the extEPG (scope = global, exported = yes) (scope = global, exported = yes)
Shared Security Import Subnet

External Subnets for the extEPG:

Allows connections to/from the
Shared Security Import Subnet: endpoints/subnets through a contract
Leaks the pcTag of the extEPG between VRFs

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 216
-
 External Classification and Route Leaking
IP Address:
Identifies remote endpoints/subnets

External Subnets for External EPG:

Required for contract purposes

• IP Address: identifies remote endpoints/subnets, must match a received route for route leaking purposes
• External Subnets for External EPG: allows packets to/from the L3out with a contract
Shared Security Import Subnet: • Shared Security Import Subnet: is always required as it leaks the extEPG pcTag to the target VRF
Leaks the pcTag/Class ID between VRFs

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 217
-
 Shared L3out – Route Leaking between VRFs (ESGs)
External Device External Device

shared-services demo

vrf-01 vrf-01

BD 192.168.150.0_24 BD 192.168.151.0_24 BD 192.168.152.0_24

Path 101/1/7 Path 102/1/7
10.237.99.233/30 10.237.99.237/30
AP
EPG pcTag:10968 EPG pcTag:12674 EPG pcTag:5468

network-
segments
192.168.150.0_24 192.168.151.0_24 192.168.152.0_24

L3out
Prefix to leak Target Tenants
vrf-01-ospf-area—0.0.0.1
Route leak
between VRFs
AP online-boutique

External Prefixes
extEPG pcTag: 41
ESG pcTag: 5490

vrf-01-all-ext-subnets all-services
C CCI I P
C CCI I P

Subnets

IP Address Scope Subnets to leak Target Tenants

0.0.0.0/1 External Subnets for the extEPG
Shared Security Import Subnet

128.0.0.0/1 External Subnets for the extEPG

Shared Security Import Subnet

Bridge Domain
Classify the external subnets and share
Subnets
the extEPG pcTag between VRFs

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 218
-
 Getting into the
weeds…!

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 219
 How does ACI Route Leaking work for EPGs…?

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 220
-
 External Classification and Route Leaking
IP Address:
Identifies remote endpoints/subnets
IP Address:
Must match a received route
for route leaking purposes

Aggregate Shared Routes:

Optional - Creates a prefix-list to
aggregate routes
Shared Route Control Subnet:
Leaks a received route to another VRF External Subnets for External EPG:
Required for contract purposes

• IP Address: identifies remote endpoints/subnets, must match a received route for route leaking purposes
• External Subnets for External EPG: allows packets to/from the L3out with a contract
Shared Security Import Subnet: • Shared Security Import Subnet: is always required as it leaks the extEPG pcTag to the target VRF
Leaks the pcTag/Class ID between VRFs
• Shared Route Control Subnet: not required when route leaking is configured under the VRF
• Aggregate Shared Routes: creates a prefix-list to aggregate routes

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 221
-
 EPG Route Leaking – L3out is the Provider
External Device External Device
Bridge Domains set to
advertise and share subnet
shared-services demo

vrf-01 vrf-01

BD 192.168.150.0_24 BD 192.168.151.0_24 BD 192.168.152.0_24

Path 101/1/7 Path 102/1/7 advertise=yes, shared=yes advertise=yes, shared=yes advertise=yes, shared=yes
10.237.99.233/30 10.237.99.237/30
AP
EPG pcTag:10968 EPG pcTag:12674 EPG pcTag:5468

192.168.150.0_24 192.168.151.0_24 192.168.152.0_24

network-segments
pcTag leaked to target C CCI I P C CCI I P C CCI I P
L3out VRF for zoning purposes
vrf-01-ospf-area—0.0.0.1

extEPG pcTag: 41
Route leak
ext-subnet-10.237.96.16
between VRFs
C CCI I P Contract
Exported
Subnets

IP Address Scope

10.237.96.16/28 External Subnets for the extEPG Cont Cont

Shared Security Import Subnet permit-to-10.237.96.16 permit-to-10.237.96.16
Shared Route Control Subnet (scope = global, exported = yes) (scope = global, exported = yes)

Shared Route Control Subnet: creates a prefix-list matching the subnet IP

address (10.237.96.16/28) which is then leaked via MP-BGP.

Shared Security Import Subnet: programs the consumer VRF with the
pcTag of the external EPG (removes blacklist)

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 222
-
 EPG Route Leaking – L3out is the Consumer
External Device External Device
Bridge Domains set to
advertise and share subnet
shared-services demo

vrf-01 vrf-01

BD 192.168.150.0_24 BD 192.168.151.0_24 BD 192.168.152.0_24

Path 101/1/7 Path 102/1/7 advertise=yes, shared=yes advertise=yes, shared=yes advertise=yes, shared=yes
10.237.99.233/30 10.237.99.237/30
AP
EPG pcTag:10968 EPG pcTag:12674 EPG pcTag:5468

192.168.150.0_24 192.168.151.0_24 192.168.152.0_24

network-segments
pcTag leaked to target C CCI I P C CCI I P C CCI I P
L3out VRF for zoning purposes
vrf-01-ospf-area—0.0.0.1 IP subnet list
advertise=yes, shared=yes
Sub Cont = no Def SVI GW

extEPG pcTag: 41
Route leak
ext-subnet-10.237.96.16
between VRFs
C CCI I P Contract
Exported
Subnets

IP Address Scope

10.237.96.16/28 External Subnets for the extEPG Cont Cont

Shared Security Import Subnet permit-to-10.237.96.16 permit-to-10.237.96.16
Shared Route Control Subnet (scope = global, exported = yes) (scope = global, exported = yes)

Shared Route Control Subnet: creates a prefix-list matching the subnet IP

address (10.237.96.16/28) which is the leaked via MP-BGP.

Shared Security Import Subnet: programs the consumer VRF with the
pcTag of the external EPG (removes blacklist)

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 223
-
 “Shared Route Control Subnet” must match a received route
External Device External Device
Bridge Domains set to
advertise and share subnet
shared-services demo

vrf-01 vrf-01

BD 192.168.150.0_24 BD 192.168.151.0_24 BD 192.168.152.0_24

Path 101/1/7 Path 102/1/7 advertise=yes, shared=yes advertise=yes, shared=yes advertise=yes, shared=yes
10.237.99.233/30 10.237.99.237/30
AP
EPG pcTag:10968 EPG pcTag:12674 EPG pcTag:5468

192.168.150.0_24 192.168.151.0_24 192.168.152.0_24

network-segments
pcTag leaked to target C CCI I P C CCI I P C CCI I P
L3out VRF for zoning purposes
vrf-01-ospf-area—0.0.0.1

extEPG pcTag: 41
Route leak
all-external-subnets
between VRFs
C CCI I P Contract
Exported
Subnets

IP Address Scope

0.0.0.0/1 External Subnets for the extEPG Cont Cont

Shared Security Import Subnet permit-to-all-external-subnets permit-to-all-external-subnets
Shared Route Control Subnet (scope = global, exported = yes) (scope = global, exported = yes)

128.0.0.0/1 External Subnets for the extEPG

Shared Security Import Subnet
Shared Route Control Subnet

Shared Route Control Subnet: creates a prefix-list matching the subnet IP

addresses (0.0.0.0/1, 128.0.0.0/1) which is then leaked via MP-BGP

THESE ROUTES WILL NEVER MATCH, AND THEREFORE WILL NEVER LEAK…!
#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 224
-
 Leaking all received routes
External Device External Device
Bridge Domains set to
advertise and share subnet
shared-services demo

vrf-01 vrf-01

BD 192.168.150.0_24 BD 192.168.151.0_24 BD 192.168.152.0_24

Path 101/1/7 Path 102/1/7 advertise=yes, shared=yes advertise=yes, shared=yes advertise=yes, shared=yes
10.237.99.233/30 10.237.99.237/30
AP
EPG pcTag:10968 EPG pcTag:12674 EPG pcTag:5468

192.168.150.0_24 192.168.151.0_24 192.168.152.0_24

network-segments
pcTag leaked to target C CCI I P C CCI I P C CCI I P
L3out VRF for zoning purposes
vrf-01-ospf-area—0.0.0.1

extEPG pcTag: 41
Route leak
all-external-subnets
between VRFs
C CCI I P Contract
Subnets Exported
IP Address Scope

0.0.0.0/1 External Subnets for the extEPG

Shared Security Import Subnet Cont Cont
Shared Route Control Subnet
permit-to-all-external-subnets permit-to-all-external-subnets
Aggregate Shared
(scope = global, exported = yes) (scope = global, exported = yes)
128.0.0.0/1 External Subnets for the extEPG
Shared Security Import Subnet
Shared Route Control Subnet
Aggregate Shared
Aggregate Shared: creates prefix-lists matching the subnet IP addresses 0.0.0.0/1 le 32 and 128.0.0.0/1 le
32, the matching routes are then leaked via MP-BGP.

Note: the scope does not match 0.0.0.0/0, thus a received default route will not leak between VRFs.
#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 225
-
 Leaking a default route
External Device External Device
Bridge Domains set to
advertise and share subnet
shared-services demo

vrf-01 vrf-01

BD 192.168.150.0_24 BD 192.168.151.0_24 BD 192.168.152.0_24

Path 101/1/7 Path 102/1/7 advertise=yes, shared=yes advertise=yes, shared=yes advertise=yes, shared=yes
10.237.99.233/30 10.237.99.237/30
AP
EPG pcTag:10968 EPG pcTag:12674 EPG pcTag:5468

192.168.150.0_24 192.168.151.0_24 192.168.152.0_24

network-segments
pcTag leaked to target C CCI I P C CCI I P C CCI I P
L3out VRF for zoning purposes
vrf-01-ospf-area—0.0.0.1

extEPG pcTag: 41
Route leak
all-external-subnets
between VRFs
C CCI I P Contract
Exported
Subnets

IP Address Scope
0.0.0.0/0 Shared Route Control Subnet
Cont Cont
0.0.0.0/1 External Subnets for the extEPG permit-to-all-external-subnets permit-to-all-external-subnets
Shared Security Import Subnet (scope = global, exported = yes) (scope = global, exported = yes)

128.0.0.0/1 External Subnets for the extEPG

Shared Security Import Subnet

Shared Route Control Subnet: creates a prefix-list matching the

subnet IP addresses (0.0.0.0/0) which is then leaked via MP-BGP

Do Not configure “External Subnets for the extEPG” for 0.0.0.0/0

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 226
-
 Why are we classifying with 0.0.0.0/1 and
128.0.0.0/1…?

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 227
-
 Non dedicated border Leafs
Upstream
network

L3Out to
external routers

hx-prod-fi-a hx-prod-fi-b

Hyperflex nodes UCS C series servers

Workloads attached
to border Leafs

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 228
-
 Shared L3out as Provider aci-dev-01-apic-01# fabric 101 show ip route vrf ssharman:vrf-01
----------------------------------------------------------------
Node 101 (aci-dev-01-leaf-101)
External Device External Device ----------------------------------------------------------------
0.0.0.0/0, ubest/mbest: 1/0
*via 10.237.99.234%shared-services:vrf-01, eth1/7, ... rwVnid: vxlan-2129920
demo !
shared-services !output truncated
Default route via shared-
vrf-01 (2555904)
vrf-01 (2129920) services:vrf-01
C CCI I P
EPG
Path 101/1/7 Path 102/1/7 Cont vzAny
10.237.99.233/30 10.237.99.237/30 (software updates)
shared-services.vrf-01-all-ext-subnets
(scope = global, imported = yes)

L3out
vrf-01-ospf-area—0.0.0.1

Route leak
Traffic routed via the external
extEPG
between VRFs network allows communication
vrf-01-all-ext-subnets between workloads in different
C CCI I P
ssharman Tenants despite no routes or
Subnets vrf-01 (3047426)
contracts in place
IP Address: 0.0.0.0/0
Cont
Scope: External Subnets for the extEPG shared-services.vrf-01-all-ext-subnets Cont EPG
Shared Route Control Subnet (scope = global, exported = yes) shared-services.vrf-01-all-ext-subnets
Shared Security Import Subnet vzAny
(scope = global, imported = yes) (software updates)
C CCI I P

aci-dev-01-apic-01# fabric 101 show ip route vrf shared-services:vrf-01 aci-dev-01-apic-01# fabric 101 show ip route vrf ssharman:vrf-01
---------------------------------------------------------------- ----------------------------------------------------------------
Node 101 (aci-dev-01-leaf-101) Node 101 (aci-dev-01-leaf-101)
---------------------------------------------------------------- ----------------------------------------------------------------
0.0.0.0/0, ubest/mbest: 1/0 0.0.0.0/0, ubest/mbest: 1/0
*via 10.237.99.234, eth1/7, ... ospf-default, type-2, tag 1 *via 10.237.99.234%shared-services:vrf-01, eth1/7, ... rwVnid: vxlan-2129920
10.237.99.160/28, ubest/mbest: 1/0, attached, direct, pervasive !
*via 10.1.176.66%overlay-1, ... static, tag 4294967292, rwVnid: vxlan-2555904
10.237.99.176/28, ubest/mbest: 1/0, attached, direct, pervasive
!output truncated
Default route via shared-
*via 10.1.176.66%overlay-1, ... static, tag 4294967292, rwVnid: vxlan-3047426
services:vrf-01
Default route to external network. Routes
-
to Tenant subnets via overlay-1 #CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 229
 Shared L3out as Consumer aci-dev-01-apic-01# fabric 101 show ip route vrf ssharman:vrf-01
----------------------------------------------------------------
Node 101 (aci-dev-01-leaf-101)
External Device External Device ----------------------------------------------------------------
0.0.0.0/0, ubest/mbest: 1/0
*via 10.237.99.234%shared-services:vrf-01, eth1/7, ... rwVnid: vxlan-2129920
demo !
shared-services !output truncated
Default route via shared-
vrf-01 (2555904)
vrf-01 (2129920) services:vrf-01
C CCI I P
EPG
Path 101/1/7 Path 102/1/7 Cont vzAny
10.237.99.233/30 10.237.99.237/30 (software updates)
permit-to-tn-demo
(scope = global, exported = yes)

L3out
vrf-01-ospf-area—0.0.0.1

Route leak
Traffic routed via the external
extEPG
between VRFs network allows communication
vrf-01-all-ext-subnets between workloads in different
C CCI I P
ssharman Tenants despite no routes or
Subnets Cont
vrf-01 (3047426)
contracts in place
IP Address: 0.0.0.0/0 permit-to-tn-demo
(scope = global, imported = yes)
Scope: External Subnets for the extEPG Cont EPG
Shared Route Control Subnet Cont
permit-to-tn-ssharman vzAny
Shared Security Import Subnet permit-to-tn-ssharman (scope = global, exported = yes) (software updates)
(scope = global, imported = yes)
C CCI I P

aci-dev-01-apic-01# fabric 101 show ip route vrf shared-services:vrf-01 aci-dev-01-apic-01# fabric 101 show ip route vrf ssharman:vrf-01
---------------------------------------------------------------- ----------------------------------------------------------------
Node 101 (aci-dev-01-leaf-101) Node 101 (aci-dev-01-leaf-101)
---------------------------------------------------------------- ----------------------------------------------------------------
0.0.0.0/0, ubest/mbest: 1/0 0.0.0.0/0, ubest/mbest: 1/0
*via 10.237.99.234, eth1/7, ... ospf-default, type-2, tag 1 *via 10.237.99.234%shared-services:vrf-01, eth1/7, ... rwVnid: vxlan-2129920
10.237.99.160/28, ubest/mbest: 1/0, attached, direct, pervasive !
*via 10.1.176.66%overlay-1, ... static, tag 4294967292, rwVnid: vxlan-2555904
10.237.99.176/28, ubest/mbest: 1/0, attached, direct, pervasive
!output truncated
Default route via shared-
*via 10.1.176.66%overlay-1, ... static, tag 4294967292, rwVnid: vxlan-3047426
services:vrf-01
Default route to external network. Routes
-
to Tenant subnets via overlay-1 #CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 230
 Recommendation

Do not use 0.0.0.0/0 in route leaking design…!

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 231
-
 Increasing Security…

-
 Where should you place your L4-7 devices…?

Do you want to use Layer 2 or Layer 3 redirects for

service insertion…?

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 259
-
 tenant “common”, “shared-services”, or in the
“workload/user” tenant…

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 260
-
 Virtual firewall deployment
shared-services
ciscolive

ftdv-04-eth5-gig-0-2
vrf: ciscolive ftdv-04-eth5-gig-0-2
vrf: ciscolive

Imported firewall
ftdv-04-eth6-gig-0-3 demo
vrf: ssharman

ftdv-04
ftdv-04-eth7-gig-0-4
ftdv-04-eth7-gig-0-4 vrf: demo
vrf: demo

Imported firewall
VRF aware firewalls defined in “shared-
services” and exported to “user” tenants

Benefits of virtual firewall / IPS

• One or more virtual firewalls exported to “user” tenants as required
• Virtual firewalls used for targeted service insertion
• Firewall throughput matches application requirements
• Firewall ruleset reduced to application requirements
• Firewall security group members pushed/pulled from APIC (where available)

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 261
-
 ACI Endpoint Update App (optional)
https://dcappcenter.cisco.com/aci-endpoint-update.html

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 262
-
 Setting up PBR to a one arm attached firewall…

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 263
-
 Redirect applied to all Filters under the Subject…
ESG ESG

frontend-svc cart-svc
ubuntu-01 C CCI I P C CCI I P ubuntu-02
192.168.150.21 192.168.151.21
Contract name typically tied to
the Provider EPG/ESG
Cont

permit-to-cart-svc

Filter/Entry name identifies

protocol, src port, and dst port
Subj Filt Entry
redirect tcp-src-any-dst-80 tcp-src-any-dst-80

Subject = redirect
Filt Entry
tcp-src-any-dst-443 tcp-src-any-dst-443

BD 6.6.6.0_24

Service Graph
Redirect

IP: 6.6.6.11
MAC: 00:50:56:a1:ac:90

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 264
-
 Setting up PBR to a two arm attached firewall…

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 265
-
 Two arm service graph – L2

The redirect policy on the The redirect policy on the

ftd-4112-cluster
Consumer Bridge domain points Provider Bridge domain points to
to the MAC on the Provider side 0008.E3D4.BBBB 0008.E3D4.E5F6 the MAC on the Consumer side
port-channel sub-interface i.e. port-channel sub-interface i.e.
0008.E3D4.E5F6 BVI pseudo-IP 0008.E3D4.BBBB
Port-channel-10 Port-channel-30

Consumer BD No IP address BD No IP address

Provider Bridge
ESG Bridge Domain Service Graph Service Graph Domain ESG
Consumer Provider
frontend-svc Bridge Domain Bridge Domain cart-svc
EPG EPG
C CCI I P C CCI I P
vlan-12 vlan-13
Consumer EPG Provider EPG
Shadow EPG Shadow EPG

Cont

permit-to-cart-svc
Cons Prov

Cons Prov

Subj Filt Entry

redirect tcp-src-any-dst-80 tcp-src-any-dst-80

Filt Entry
tcp-src-any-dst-443 tcp-src-any-dst-443

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 266
-
 Do not use Two Arm with L3 redirect…

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 267
-
 Two arm service graph – L3 Cons to Prov
> show route

Firewall static
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
<output truncated>
routes
Gateway of last resort is not set

ftd-4112-cluster C 6.6.6.0 255.255.255.0 is directly connected, Port-channel10-vlan-12

L 6.6.6.10 255.255.255.255
is directly connected, Port-channel10-vlan-12
C 7.7.7.0 255.255.255.0 is directly connected, Port-Cannel30-vlan-13
L 7.7.7.10 255.255.255.255 is directly connected, Port-Cannel30-vlan-13
S 192.168.151.0 255.255.255.0 [1/0] via 6.6.6.1, Port-channel10-vlan-12
Port-channel-10 Port-channel-30 S 192.168.152.0 255.255.255.0 [1/0] via 7.7.7.1, Port-channel30-vlan-13

BD 192.168.152.0_24 Consumer BD 6.6.6.0_24 BD 7.7.7.0_24

Provider Bridge BD 192.168.152.0_24

ESG Bridge Domain Service Graph Service Graph Domain ESG

Consumer Provider
frontend-svc Bridge Domain Bridge Domain cart-svc
EPG EPG
C CCI I P C CCI I P
vlan-12 vlan-13
Consumer EPG Provider EPG
Shadow EPG Shadow EPG

Cont

permit-to-cart-svc
Cons Prov

Cons Prov

Subj Filt Entry

redirect tcp-src-any-dst-80 tcp-src-any-dst-80

Filt Entry
tcp-src-any-dst-443 tcp-src-any-dst-443

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 268
-
 Two arm service graph – L3 Prov to Cons
> show route

Firewall static
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
<output truncated>
routes
Gateway of last resort is not set

ftd-4112-cluster C 6.6.6.0 255.255.255.0 is directly connected, Port-channel10-vlan-12

L 6.6.6.10 255.255.255.255
is directly connected, Port-channel10-vlan-12
C 7.7.7.0 255.255.255.0 is directly connected, Port-Cannel30-vlan-13
L 7.7.7.10 255.255.255.255 is directly connected, Port-Cannel30-vlan-13
S 192.168.151.0 255.255.255.0 [1/0] via 6.6.6.1, Port-channel10-vlan-12
Port-channel-10 Port-channel-30 S 192.168.152.0 255.255.255.0 [1/0] via 7.7.7.1, Port-channel30-vlan-13

BD 192.168.151.0_24 Consumer BD 6.6.6.0_24 BD 7.7.7.0_24

Provider Bridge BD 192.168.152.0_24

ESG Bridge Domain Service Graph Service Graph Domain ESG

Consumer Provider
frontend-svc Bridge Domain Bridge Domain cart-svc
EPG EPG
C CCI I P C CCI I P
vlan-12 vlan-13
Consumer EPG Provider EPG
Shadow EPG Shadow EPG

Cont
Traffic received on the Firewall consumer interface.
permit-to-cart-svc
The Firewall has a static route pointing back out of the
Cons Prov

incoming interface, therefore traffic does not traverse Cons Prov

the consumer/provider interfaces.

Subj Filt Entry

redirect tcp-src-any-dst-80 tcp-src-any-dst-80

Filt Entry
tcp-src-any-dst-443 tcp-src-any-dst-443

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 269
-
 Layer 2 vs Layer 3 firewall insertion
Layer 2 – two arm attached • Layer 3 - one arm attached
• Transparent IPS mode • Simple “default” routing
• Slightly higher throughput • Easier installation method
• No routing • Used by more than 95% of customers

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 270
-
 Using Service Graphs for Security Enforcement
and/or Application Dependency Mapping…

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 274
-
 EPU retrieves dynamic
endpoint information from

Dynamic Endpoint Updates APIC APIC and updates FMC

External Device External Device FMC EPU

shared-services FMC updates FTD with

vrf-01 dynamic endpoint information
extEPG IP Address: 0.0.0.0/1
128.0.0.0/1

Scope: External Subnets for the extEPG

Shared Security Import Subnet

C CCI I P

Route leak
between VRFs
demo

vrf-01
AP online-boutique
Cont
C CCI I P
Name: permit-to-online-boutique-frontend-services ESG frontend-service
Scope: global
Exported: yes (to shared-services)

Subject: tcp C CCI I P

Service Graph: yes – redirect-to-ftdv-02-eth7-gig-0-4
Stateful: no
Filter: tcp-src-any-dst-80
tcp-src-any-dst-8080
Cont
Name: permit-to-online-boutique-backend-services
Scope: global
Exported: no

Subject: permit-any
Service Graph: yes – redirect-to-ftdv-02-eth7-gig-0-4
Stateful: no
Cont Filter: permit-any
BD 6.6.6.0_24
Name: intra-esg-online-boutique-backend-services
Service Graph Scope: vrf
Redirect Exported: no
C CCI I P
Subject: permit-any ESG backend-services
Service Graph: yes – redirect-to-ftdv-02-eth7-gig-0-4
Stateful: no
Filter: permit-any
C CCI I P

Intra ESG isolation = enforced

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 275
-
 EPU retrieves dynamic

Targeted Flow Analysis endpoint information from

APIC and updates FMC

External Device External Device APIC

shared-services
FMC updates FTD with
vrf-01
dynamic endpoint information
extEPG IP Address: 0.0.0.0/1 FMC EPU
128.0.0.0/1

Scope: External Subnets for the extEPG

Shared Security Import Subnet

C CCI I P

Syslog
Route leak
between VRFs
demo

vrf-01
AP online-boutique
FTD Syslog flow information
Cont
C CCI I P for manual ADM
Name: permit-to-online-boutique-frontend-services ESG frontend-service
Scope: global
Exported: yes (to shared-services)

Subject: tcp C CCI I P

Service Graph: yes – redirect-to-ftdv-02-eth7-gig-0-4
Stateful: no
Filter: tcp-src-any-dst-80

Cont

FTD configured to generate Syslog Name:

Scope:
permit-to-online-boutique-backend-services
global
messages for all flows Exported: no

Subject: permit-any
Service Graph: yes – redirect-to-ftdv-02-eth7-gig-0-4
Stateful: no
Cont Filter: permit-any
BD 6.6.6.0_24
Name: intra-esg-online-boutique-backend-services
Service Graph Scope: vrf
Redirect Exported: no
C CCI I P
Subject: permit-any ESG backend-services
Service Graph: yes – redirect-to-ftdv-02-eth7-gig-0-4
Stateful: no
Filter: permit-any
C CCI I P

Intra ESG isolation = enforced

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 276
-
 EPU retrieves dynamic

Targeted Flow Analysis and Enforcement endpoint information from

APIC and updates FMC

External Device External Device APIC

shared-services
FMC updates FTD with
vrf-01
dynamic endpoint information
extEPG IP Address: 0.0.0.0/1 FMC EPU
128.0.0.0/1

Scope: External Subnets for the extEPG

CSW updates FMC with

Shared Security Import Subnet

C CCI I P dynamic firewall policies

Cisco Secure Workload Syslog
Route leak
between VRFs
demo

vrf-01
AP online-boutique
FTD updates CSW and Splunk
Cont
C CCI I P with flow information for ADM
Name: permit-to-online-boutique-frontend-services ESG frontend-service
Scope: global
Exported: yes (to shared-services)

Subject: tcp C CCI I P

Service Graph: yes – redirect-to-ftdv-02-eth7-gig-0-4
Stateful: no
Filter: tcp-src-any-dst-80

Cont

FTD configured to generate NetFlow records Name:

Scope:
permit-to-online-boutique-backend-services
global
and Syslog messages for all flows Exported: no

Subject: permit-any
Service Graph: yes – redirect-to-ftdv-02-eth7-gig-0-4
Stateful: no
Cont Filter: permit-any
BD 6.6.6.0_24
Name: intra-esg-online-boutique-backend-services
Service Graph Scope: vrf
Redirect Exported: no
C CCI I P
Subject: permit-any ESG backend-services
Service Graph: yes – redirect-to-ftdv-02-eth7-gig-0-4
Stateful: no
Filter: permit-any
C CCI I P

Intra ESG isolation = enforced

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 277
-
 Tightening
Security…

-
 Step 1: Assign Endpoints to the “correct” ESG…

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 279
-
 Assign endpoints by Tagging the endpoints to the
correct group…
demo

vrf-01

BD 192.168.150.0_24 BD 192.168.151.0_24 BD 192.168.152.0_24

L3Out
AP
EPG EPG EPG extEPG
Network-
segments

192.168.150.0_24 192.168.151.0_24 192.168.152.0_24 0.0.0.0/1

dynamic (P,S) vlans dynamic (P,S) vlans dynamic (P,S) vlans 128.0.0.0/1
C CCI I P C CCI I P C CCI I P AP epg-matched-esg C CCI I P

ESG

network-segments

C CCI I P

vzAny as a contract consumer defines

All applications initially provide the same
that all EPGs, ESGs, extEPGs are
contract to vzAny. This maintains open
C CCI P

consumers of the same contract

vzAny

communication between applications All EPGs, ESGs, extEPGs

C CCI P

Cont
permit-to-all-applications
A single contract allows workloads to
move between ESGs without breaking
network forwarding

AP application-01 AP application-02 AP application-03

C CCI I P C CCI I P C CCI I P
ESG all-services ESG all-services ESG all-services

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 280
-
 Step 2: Decide how to tighten security…

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 281
-
 Application security options…
Cisco Secure
Workload

Split apps into discreet

Tenants/VRFs – repeat

Discreet application Security groups for

tier ESGs each application tier?

Intra ESG with SG for Control E/W flows

L4-7 enforcement within the application?

Leverage ESG for Discover E/W flows

Security

Application Mapping within the application?

Service Graphs (SG) Insert L4-7 device

between applications between applications?

Stateful contracts Control the direction of

between applications session establishment?

Restricted contracts Which application ports

between applications need exposing?

Open contracts Which applications can

between applications communicate?

Open contracts to
vzAny

Application Knowledge
#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 282
-
 Remember this from a little earlier…

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 283
-
 permit any/any
ESG ESG

frontend-svc cart-svc
ubuntu-01 C CCI I P C CCI I P ubuntu-02
192.168.150.21 192.168.151.21

Cont

permit-any

Subj Filt Entry

permit-any permit-any unspecified

# netcat –p [any] ubuntu-02 [any] # netcat –l [any]

Communication allowed to/from any

ubuntu-01 protocol/port in both directions ubuntu-02
192.168.150.21 192.168.151.21
# netcat –l [any] # netcat –p [any] ubuntu-01 [any]

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 284
-
 permit any/any with vzAny
demo

vrf-01

ESG
Cont Subj
frontend-svc permit-to-frontend-svc permit-src-any-dst-any
ubuntu-01 C CCI I P

192.168.150.21
Communication allowed on any
protocol/port to the frontend-svc Filt

C CCI P permit-src-any-dst-any
EPG
Communication allowed on any vzAny
protocol/port from the frontend-svc
C CCI P
Entry

permit-src-any-dst-any
C CCI I P C CCI I P C CCI I P
extEPG EPG ESG

All extEPGs All EPGs All ESGs

any external any EPG any ESG

device attached device attached device

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 285
-
 demo

vrf-01

AP AP AP AP AP
C CCI I P C CCI I P C CCI I P C CCI I P

application-04
application-01

application-02

application-03
C CCI I P

application-05
ESG ESG ESG ESG ESG
all-services all-services all-services all-services all-services
C CCI I P C CCI I P C CCI I P C CCI I P C CCI I P

Cont Cont Cont Cont

permit-to-all-applications permit-to-application-03 permit-to-application-04 permit-to-application-05

C CCI P
vzAny

All EPGs, ESGs, extEPGs

C CCI P

Consumer Contract Filter Provider Notes

vzAny permit-to-all-applications permit-src-any-dst-any application-01 vzAny with permit-src-any-dst-any allows

(all EPGs/ESGs/extEPGs) bi-directional communication on any port
between all applications
vzAny permit-to-all-applications permit-src-any-dst-any application-02 vzAny with permit-src-any-dst-any allows
(all EPGs/ESGs/extEPGs) bi-directional communication on any port
between all applications
application-04 permit-to-application-03 permit-src-any-dst-any application-03 Bi-directional communication on any port
between application-04 and application-03
application-03 permit-to-application-04 permit-tcp-src-any-dst-443 application-04 Communication from application-03 and
application-05 application-05
application-04 permit-to-application-05 permit-src-any-dst-any application-05 Bi-directional communication on any port
(Service graph to FTD for inter via firewall between application-04 and
and intra application flows) application-05

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 286
-
 Let’s tighten the contract to our online-boutique
application…

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 287
-
 Tighten access to our online-boutique application…
External Device External Device

shared-services

vrf-01
extEPG IP Address: 0.0.0.0/1 Cont
128.0.0.0/1
Name: permit-to-tn-demo-online-boutique
Imported: yes (from demo)
Scope: External Subnets for the extEPG
Shared Route Control Subnet*
Subject: tcp
Shared Security Import Subnet
Stateful: yes
C CCI I P Filter: tcp-src-any-dst-80
tcp-src-any-dst-8080

Route leak
demo between VRFs
vrf-01
Cont Cont

Name: permit-to-core-services Name: permit-from-core-services

Imported: yes (from shared-services) Imported: yes (from shared-services)
Contract Exported
Subject: udp Subject: tcp
Stateful: no Stateful: no
Filter: udp-src-any-dst-53 Filter: tcp-src-22-dst-any AP epg-matched-esg “outside” to application
udp-src-any-dst-123
ESG
Cont
requires ports TCP
80/8080
Name: permit-to-tn-demo-online-boutique
Exported: yes (to shared-services)
C CCI I P network-segments

Contracts exported from EPG

C CCI I P
Subject: tcp
Stateful: yes
“shared-services” vzAny Filter: tcp-src-any-dst-80
tcp-src-any-dst-8080
C CCI I P

Tighten access to our

BD 192.168.150.0_24 BD 192.168.151.0_24 BD 192.168.152.0_24
application
vzAny allows access AP C CCI
EPG
I P C CCI
EPG
I P C CCI
EPG
I P AP online-boutique

to/from “core-services”
C CCI I P
Network-
segments

192.168.150.0_24 192.168.151.0_24 192.168.152.0_24

dynamic (P,S) vlans dynamic (P,S) vlans dynamic (P,S) vlans ESG all-services
Intra EPG = Unenforced Intra EPG = Unenforced Intra EPG = Unenforced
C CCI I P C CCI I P C CCI I P

demo:online-boutique

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 288
-
 Automation Blueprints…

-
 Application tiers across subnets
Application Centric Blueprint #1 – ESG “wrapper” for all services
demo

vrf-01

Consumers
Single security zone for
all application services
AP online-boutique
C CCI I P
ESG all-services

frontend

checkout

adservice recommendation shipping email

payment

product catalog cart currency

Redis cache

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 290
-
 Application tiers across subnets
Application Centric Blueprint #2 – Intra ESG Isolation
demo

vrf-01

Consumers
Firewall/IPS
Single isolated security zone
for all application services
AP online-boutique
C CCI I P
ESG all-services

Intra ESG contract with

frontend Service Graph redirect
Protect against application to Firewall/IPS
vulnerabilities such as Log4j
checkout

adservice recommendation shipping email

payment

product catalog cart currency

Redis cache

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 291
-
 Application tiers across subnets
Application Centric Blueprint #3 – Dedicated AP/ESG for backend database
demo

vrf-01

Consumers

AP online-boutique
C CCI I P
ESG all-services

frontend

checkout

adservice recommendation shipping email

payment

product catalog cart currency

C CCI I P

AP databases
C CCI I P
ESG redis
Dedicated Application Profile
Redis cache
and ESG (with contract) for
database services

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 292
-
 Application tiers across subnets
Application Centric Blueprint #4 – Inbound firewall/IPS + backend contract
demo

vrf-01

Consumers
Inbound firewall/IPS

AP online-boutique
C CCI I P
ESG all-services
Inbound Firewall between
Consumers and Application
frontend

checkout

adservice recommendation shipping email

payment

product catalog cart currency

C CCI I P

AP databases
C CCI I P
ESG redis
Dedicated Application Profile
Redis cache
and ESG (with contract) for
database services

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 293
-
 Application tiers across subnets
Application Centric Blueprint #5 – Inbound firewall/IPS + backend firewall/IPS
demo

vrf-01

Consumers
Inbound firewall

AP online-boutique
C CCI I P
ESG all-services
Inbound Firewall between
Consumers and Application
frontend

checkout

adservice recommendation shipping email

payment

product catalog cart currency

C CCI I P

AP databases
C CCI I P
ESG redis

Backend Firewall between

Redis cache
Application and Database
Database firewall

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 294
-
 Application tiers across subnets Requires application
Application Centric Blueprint #6 – ESG per application tier dependency map
demo

vrf-01

Consumers
Single security zone for
each application service
AP online-boutique
C CCI I P C CCI I P
ESG ESG
frontend checkout

C CCI I P C CCI I P

C CCI I P C CCI I P C CCI I P C CCI I P

ESG ESG ESG ESG
adservice recommendation payment email

C CCI I P C CCI I P C CCI I P C CCI I P

C CCI I P C CCI I P C CCI I P C CCI I P

ESG ESG ESG ESG
product
cart shipping currency
catalogue

C CCI I P C CCI I P C CCI I P C CCI I P

C CCI I P
ESG
redis

C CCI I P

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 295
-
 Application tiers across subnets
Application Centric Blueprint #7 – Dedicated AP/ESG for backend database
demo

vrf-01

Consumers
Single security zone for
each application service
AP online-boutique
C CCI I P C CCI I P
ESG ESG
frontend checkout

C CCI I P C CCI I P

C CCI I P C CCI I P C CCI I P C CCI I P

ESG ESG ESG ESG
adservice recommendation payment email

C CCI I P C CCI I P C CCI I P C CCI I P

C CCI I P C CCI I P C CCI I P C CCI I P

ESG ESG ESG ESG
product
cart shipping currency
catalogue

C CCI I P C CCI I P C CCI I P C CCI I P

AP databases
C CCI I P
ESG
Dedicated Application Profile
redis
and ESG (with contract) for
C CCI I P database services

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 296
-
 Application tiers across subnets
Application Centric Blueprint #8 – ESG per application tier + frontend firewall/IPS
demo

vrf-01

Consumers
Frontend firewall/IPS
Single security zone for
each application service Inbound Firewall between
Consumers and application
“frontend”
AP online-boutique
C CCI I P C CCI I P
ESG ESG
frontend checkout

C CCI I P C CCI I P

C CCI I P C CCI I P C CCI I P C CCI I P

ESG ESG ESG ESG
adservice recommendation payment email

C CCI I P C CCI I P C CCI I P C CCI I P

C CCI I P C CCI I P C CCI I P C CCI I P

ESG ESG ESG ESG
product
cart shipping currency
catalogue

C CCI I P C CCI I P C CCI I P C CCI I P

AP databases
C CCI I P
ESG
Dedicated Application Profile
redis
and ESG (with contract) for
C CCI I P database services

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 297
-
 Application tiers across subnets
Application Centric Blueprint #9 – ESG per application tier + frontend firewall/IPS + backend firewall/IPS
demo

vrf-01

Consumers
Inbound firewall
Single security zone for
each application service Inbound Firewall between
Consumers and application
“frontend”
AP online-boutique
C CCI I P C CCI I P
ESG ESG
frontend checkout

C CCI I P C CCI I P

C CCI I P C CCI I P C CCI I P C CCI I P

ESG ESG ESG ESG
adservice recommendation payment email

C CCI I P C CCI I P C CCI I P C CCI I P

C CCI I P C CCI I P C CCI I P C CCI I P

ESG ESG ESG ESG
product
cart shipping currency
catalogue

C CCI I P C CCI I P C CCI I P C CCI I P

AP databases
C CCI I P
ESG
Backend Firewall between
redis
Application and Database
C CCI I P Database firewall

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 298
-
 Application tiers across subnets
Application Centric Blueprint #10 – ESG per application tier + frontend, backend, and payment firewall/IPS
demo

vrf-01

Consumers
Inbound firewall
Single security zone for
each application service Inbound Firewall between
Consumers and application
“frontend”
AP online-boutique
C CCI I P C CCI I P
ESG ESG
frontend checkout

C CCI I P C CCI I P Payment firewall Firewall between

“checkout” and “payment”
C CCI I P C CCI I P C CCI I P C CCI I P
ESG ESG ESG ESG
adservice recommendation payment email

C CCI I P C CCI I P C CCI I P C CCI I P

C CCI I P C CCI I P C CCI I P C CCI I P

ESG ESG ESG ESG
product
cart shipping currency
catalogue

C CCI I P C CCI I P C CCI I P C CCI I P

AP databases
C CCI I P
ESG
Backend Firewall between
redis
Application and Database
C CCI I P Database firewall

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 299
-
 The ultimate aim is to provide a fully consumable
fabric where resources are automated on
demand…

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 300
-
 Example Internal Private Cloud Design – shared subnet(s)
External Device External Device

Bridge Domain in the

shared-services common “common” tenant can be
vrf-01 common.vrf-01 shared across multiple tenants
extEPG IP Address: 0.0.0.0/1

Route leak
128.0.0.0/1

Tenant VRF not required as

BD Name: 10.1.0.0_16
Scope: External Subnets for the extEPG
Shared Route Control Subnet*
between VRFs Gateway: 10.1.0.1/16

the “landing zone” EPG is

Shared Between VRFs: Yes
Shared Security Import Subnet Advertise Externally: Yes

mapped to the BD in C CCI I P

“common”
demo
Endpoints security policy
vrf-01
Contracts exported to moved to application ESGs
“shared-services” based on tag policy
AP application-1
Cont Cont
C CCI I P
permit-to-core-services permit-from-core-services Cont ESG all-services
(exported from shared-services) (exported from shared-services) permit-to-tn-demo-application-1
(exported to shared-services)

Contracts exported from

C CCI I P
EPG
AP network-segments
“shared-services” vzAny
EPG
C CCI I P AP application-2
C CCI I P
C CCI I P Cont ESG all-services
permit-to-tn-demo-application-2
(exported to shared-services)

vzAny allows access

to/from “core-services” C CCI I P AP application-3
C CCI I P
Cont ESG all-services
permit-to-tn-demo-application-3
(exported to shared-services)

vzAny cannot be a
provider for shared Application endpoints deployed to an EPG
services “landing zone” in “enforced” mode to
prevent E/W traffic inside both the
hypervisor and the network
#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 301
-
 Example Internal Private Cloud Design – Auto Cleanup
Isolation External Device External Device

Bridge Domain in the

shared-services common “common” tenant can be
vrf-01 common.vrf-01 shared across multiple tenants
extEPG IP Address: 0.0.0.0/1

Route leak
128.0.0.0/1

Tenant VRF not required as

BD Name: 10.1.0.0_16
Scope: External Subnets for the extEPG
Shared Route Control Subnet*
between VRFs Gateway: 10.1.0.1/16

the “landing zone” EPG is

Shared Between VRFs: Yes
Shared Security Import Subnet Advertise Externally: Yes

mapped to the BD in C CCI I P

“common”
demo
Endpoints security policy
vrf-01
Contracts exported to moved to application ESGs
“shared-services” based on tag policy
AP application-1
Cont Cont
C CCI I P
permit-to-core-services permit-from-core-services Cont ESG all-services
(exported from shared-services) (exported from shared-services) permit-to-tn-demo-application-1
(exported to shared-services)

Contracts exported from

C CCI I P
EPG
AP network-segments
“shared-services” vzAny
EPG
C CCI I P AP application-2
C CCI I P
C CCI I P Cont ESG all-services
permit-to-tn-demo-application-2
(exported to shared-services)

vzAny allows access

to/from “core-services” C CCI I P AP application-3
C CCI I P
Cont ESG all-services
permit-to-tn-demo-application-3
(exported to shared-services)

vzAny cannot be a
provider for shared Application endpoints deployed to an EPG
services “landing zone” in “enforced” mode to
prevent E/W traffic inside both the
hypervisor and the network
#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 302
-
 Wrapping up…

-
 Select one or more Design Patterns…
Carefully consider the use of:
• Tenant “common”
• Using a “shared services” tenant
• vzAny
• Dedicated border Leafs (recommended)
• Contract scopes
• External EPG with the classifier 0.0.0.0/0

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 304
-
 Benefits of Shared Service model…
• Looks and feels like a Public Cloud model of working
• Network team maintains control of North / South route peering
• Network team maintains control of Inter VRF route leaking
• Each Tenant can control their own CIDR range
• Each Tenant can control their own security rules
• Each Tenant can have private (non routable subnets)
• Security services can be easily inserted in the Tenants
• Do not use 0.0.0.0/0 as the extEPG classifier in a shared model

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 305
-
 Implement ESG “wrappers”…
Wrapping applications into ESGs provides the following benefits
for both virtual and physical workloads:
• Improved application visibility
• Improved auditing capabilities
• Improved troubleshooting
• Intelligent service insertion
• Security tied applications rather than network segments
• Reduce the reliance on monolithic physical security devices

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 306
-
 Automation Considerations…
• A simple consumption model is everything
• Single API for all networking functions
• Application security requirements should be declared to the infrastructure
• Add virtual application firewalls to deployments if required
• Large physical monolithic firewalls are useful at network boundaries, however they should
only provide broad security rules
• Remove unnecessary overlay networks that add layers of complexity

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 307
-
 Getting started resources
• Visual Studio Code with extensions
• Yaml
• Indent Rainbow
• Hashi Terraform

• https://netascode.cisco.com/solutions/aci/terraform/overview

• https://developer.cisco.com/docs/nexus-as-code/introduction/

• https://github.com/netascode/terraform-aci-nac-aci/tree/main

• https://github.com/netascode/nac-aci-simple-example

• https://github.com/netascode/nac-aci-comprehensive-example

• https://github.com/spsharman/CiscoLive2024

• https://github.com/spsharman/aci-prod

• https://github.com/spsharman/aci-dev-01

• https://tl10k.dev/categories/terraform/nexus-as-code-architecture/

#CiscoLiveAPJC CISCOU-2033 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 308
-
 ESG Design Guide
https://www.cisco.com/c/en/us/td/docs/dcn/whitepapers/cisco-aci-esg-design-guide.html?cachemode=refresh

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 309
-
 Now available on dCloud
https://dcloud2-sjc.cisco.com/content/demo/333928?returnPathTitleKey=content-view

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 310
-
 Try the Walk in Lab (LABDCN-2287) in the World of
Solutions…

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 311
-
 Complete Your Session Evaluations

Complete a minimum of 4 session surveys and the Overall Event Survey to

claim a Cisco Live T-Shirt.

Complete your surveys in the Cisco Live mobile app.

#CiscoLiveAPJC BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 312
-
 Continue
your education • Visit the Cisco Showcase
for related demos

LABDCN-2287 – ACI Segmentation… • Book your one-on-one

BRKDCN-2634 – Deploying EVPN G/W… Meet the Engineer meeting
BRKDCN-2673 – Nexus-as-Code…
BRKDCN-2910 – Upgrading ACI…
BRKDCN-2949 – ACI Multi-Pod… • Attend the interactive education
BRKDCN-2980 – ACI Multi-Site… with DevNet, Capture the Flag,
BRKDCN-3900 – ACI Forwarding… and Walk-in Labs
BRKDCN-3982 – ACI PBR Deep Dive…

• Visit the On-Demand Library

for more sessions at
www.CiscoLive.com/on-demand

Contact me at: ssharman@cisco.com

BRKDCN-2984 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 313
-
 Thank you

-
#CiscoLiveAPJC
 #CiscoLiveAPJC
-