Introduction to Cisco ACI

Audio Thank you for joining. Please mute your audio.

If you have any questions, please ask them in

Questions the Q&A Panel.

Timing The workshop will start at 9:10 EEST time.

You will receive the slides in the PDF format

Slides after taking the training survey.
Introduction to Cisco ACI
3-Day Training

Ratko Perlic
ratko.perlic@flintmail.com
§ Improve interaction
§ Enables Q&A
§ Engage everybody

https://tinyl.io/4TW4
Day 3 Agenda
§ Endpoint Manager
§ Contracts
§ Service Graph
§ Multicast
§ ACI L2 Extension and STP
§ L3Out Datapath
§ Backup & Maintenance
§ ACI Deployment Models
Endpoint Manager
Endpoint Manager (EPM) Process Architecture
Oracle ZM
on
e cti QT
CP
o nn Co
PC nn
TC ec
tio
Q n
ZM
Spine
SUP SUP
COOP COOP
ZMQ TCP Connection Objstore
Objstore
(vPC sync)
EPM EPM

LC LC
EPMC EPMC

ASIC ASIC ASIC ASIC

Leaf1 Leaf2
• Start by moquery or show endpoint on the APIC (classes epmMacEp, epmIpEp)
• find if and where the endpoint is known
• Repeat few times to see if it is stable

• On leaf, use show system internal epm endpoint [ip | mac]

• Repeat few times to see if it is known and stable
• Get EPM(C) traces on leaf Save all EPM files and eventually TechSupport
• show endpoint [detail]

Basic command; shows encap VLAN and interface

one (or two) line per EP—good for grepping
• show system internal epm endpoint [ip|mac] [x.x.x.x|xxxx.xxxx.xxxx]
Useful most of the time
shows more detail, internal vlan etc.
• vsh_lc –c ’Show system internal epmc endpoint [ip|mac] [x.x.x.x|xxxx.xxxx.xxxx]’
Most detailed
closest to the hardware
shows learned source, aging information etc.
 Show Endpoint APIC and Leaf
apic1# show endpoint ip 10.200.2.105
Legends:
(P):Primary VLAN
(S):Secondary VLAN

Dynamic Endpoints:
Tenant : DC
Application : App
AEPg : EPG2
End Point MAC IP Address Node Interface Encap Multicast Address
----------------- ---------------- ---------- ------------------- -------------- ---------------
00:00:00:01:10:02 10.200.2.105 101 eth1/3 vlan-1002 not-applicable

Total Dynamic Endpoints: 1

Total Static Endpoints: 0
leaf1# show endpoint ip 10.200.1.98 detail
Legend:
O - peer-attached H - vtep a - locally-aged S - static
V - vpc-attached p - peer-aged L - local M - span
s - static-arp B - bounce
+-----------------------------------+---------------+-----------------+--------------+-------------+-----------------------+
VLAN/ Encap MAC Address MAC Info/ Interface Endpoint Group
Domain VLAN IP Address IP Info Info
+-----------------------------------+---------------+-----------------+--------------+-------------+-----------------------+
12 vlan-2034 0050.56a4.0aa1 L eth1/9 DC:App:EPG1
DC:DC vlan-2034 10.200.1.98 L eth1/9
 Dissecting EPM Command
Key endpoint information: 3 options:
1. MAC only – local and remote EP in layer-2 mode BD
2. Mac + x IP (0<x<1024) – local EP in BD with unicast routing enabled
3. IP only – remote EP in BD with unicast routing enabled
VRF VNID – unique ID for a VRF – used in VXLAN header for
routed packet
pod2-leaf1# show system internal epm endpoint mac 0050.568a.21ed
Vlan ID – PI internal VLAN where the EP is learned with its VNID
MAC : 0050.568a.21ed ::: Num IPs : 1
typically the FD_VLAN (internal value for encap VLAN in EPG)
IP# 0 : 1.1.1.15 ::: IP# 0 flags :
Vlan id : 120 ::: Vlan vnid : 8904 ::: VRF name : DC:span BD_VNID = VNID for BD parent of the EPG
BD vnid : 16678779 ::: VRF vnid : 2949120 Used in eVXLAN header for L2 packet in the fabric
Phy If : 0x1a02e000 ::: Tunnel If : 0
Interface : Ethernet1/47
Flags : 0x80004c04 ::: sclass : 16386 ::: Ref count : 5
EP Create Timestamp : 09/12/2017 09:07:39.810800 Interface information: learned source interface, either a tunnel (from
EP Update Timestamp : 09/13/2017 09:59:21.308830 fabric) or a physical interface for locally learned EP
EP Flags : local|IP|MAC|sclass|timer|
::::
Sclass – the unique EPG identifier in the VRF (same sclass can be
reused by different EPG if they are in different VRF)
Sclass below 16k are Global scope (typically used for Inter-VRF)
 Dissecting EPMC Command
Learn Source can be : Aging line:
NS : learned by NorthStar ASIC Timer-type – HT (Host Tracker), Age, Bounce or Hold
EPM : learned from EPM – likely a epsync from vpc peer Timeout-left – count down timer till 0 or refreshed
Hit-bit – flag set when we see hit bit in the entry set in hw ASIC, reset when timer is refreshed
Timer-reset count – how many times that entry was refreshed (without being deleted)

pod2-leaf1# vsh_lc -c 'show system internal epmc endpoint mac 0050.568a.5429'

MAC : 0050.568a.5429 ::: Num IPs : 0
Vlan id : 135 ::: Vlan vnid : 9306112 ::: BD vnid : 16383903
Encap vlan : VXLAN/9306112
VRF name : DC:DC ::: VRF vnid : 2097153
phy if : 0 ::: tunnel if : 0x18010007 ::: Interface : Tunnel7
Flags : 0x80004815
Ref count : 4 ::: sclass : 32770
Timestamp : 01/22/1970 22:27:20.960287
last mv timestamp 01/01/1970 00:00:00.000000 ::: ep move count : 0
last loop_detection_ts 01/01/1970 00:00:00.000000
previous if : 0 ::: loop detection count : 0
Learn Src: EPM
EP Flags : local|vPC|locally-aged|MAC|sclass|timer|
Aging: Timer-type : HT ::: Timeout-left : 619 ::: Hit-bit : No ::: Timer-reset count : 0
PD handles:
Bcm l2 hit-bit : No
[L2]: Asic : NS ::: ADJ : 0x13 ::: LST SA : 0xd55 ::: LST DA : 0xd55 ::: GST ING : 0x8ee ::: BCM : No
<detail> SDB Data:
mod 4 ::: port 7
Hw info pointer to various hardware table in Broadcom or Cisco ASIC
• EPM (Endpoint Manager) and EPMC (EPM Client) keep complete log of all EP activity.
• In a production fabric those logs may wrap very quickly
• around 1 GB of trace is kept in a rotating log for each leaf
• Location of those logs on each leaf :
/var/log/dme/log/epm*

• Best practice is to copy and store those files as quick as possible after an event you want to
analyze
mkdir /tmp/epm-log-20210302
cp /var/log/dme/epm* /tmp/epm-log-20210302
 Endpoint Move to Remote Leaf

5b 3 2
5a 5c

Leaf3 4 Leaf1 Leaf2

1
0. Whole fabric knows that EP is connected to Leaf1
1. EP moves from Leaf1 to Leaf2 (for example vMotion)
2. Leaf2 sees EP and updates internal DB and COOP DB on Spine
3. COOP is updated and tells Leaf1 than EP is now on Leaf2
4. Leaf1 creates a bounce entry. Bounce entry remains for a configurable value which should be greater than
remote endpoint aging
5. Packets from Leaf3 (who may have older EP entry pointing to Leaf1) will reach Leaf1, that bounces them to
Leaf2; returning packets will properly set EP entry on Leaf3
 VM Moved from One Leaf to Another Leaf
(Original Leaf Shown)—EPMC Bounce Entry
BEFORE AFTER

module-1# show system internal epmc endpoint ip module-1# show system internal epmc endpoint ip
2.2.2.200 2.2.2.200
MAC : 0050.56bf.1082 ::: Num IPs : 1 MAC : 0000.0000.0000 ::: Num IPs : 1
IP# 0 : 2.2.2.200 ::: IP# 0 flags : IP# 0 : 2.2.2.200 ::: IP# 0 flags :
Vlan id : 15 ::: Vlan vnid : 8292 ::: BD vnid : Vlan id : 0 ::: Vlan vnid : 0 ::: BD vnid : 0
16318375 VRF name : aobst:Network1 ::: VRF vnid : 2097153
Encap vlan : 802.1Q/2900 phy if : 0 ::: tunnel if : 0x18010005 :::
VRF name : aobst:Network1 ::: VRF vnid : 2097153 Interface : Tunnel5
phy if : 0x1a003000 ::: tunnel if : 0 ::: Interface Flags : 0x80004408
: Ethernet1/4 Ref count : 3 ::: sclass : 16389
Flags : 0x80004c04 Timestamp : 01/01/1970 10:44:59.504552
Ref count : 5 ::: sclass : 16389 last mv timestamp 01/01/1970 10:00:00.000000 :::
Timestamp : 01/01/1970 10:42:25.275539 ep move count : 0
last mv timestamp 01/01/1970 10:00:00.000000 ::: ep last loop_detection_ts 01/01/1970 10:00:00.000000
move count : 0 previous if : 0x18010005 ::: loop detection count
last loop_detection_ts 01/01/1970 10:00:00.000000 : 0
previous if : 0 ::: loop detection count : 0 EP Flags : bounce, IP,class-set,timer,
EP Flags : local,IP,MAC,class-set,timer, Aging:Timer-type : Bounce timeout ::: Timeout-
Aging:Timer-type : Host-tracker timeout ::: Timeout- left : 604 ::: Hit-bit : No ::: Timer-reset count :
left : 72 ::: Hit-bit : Yes ::: Timer-reset count : 3 0
...
 egrep "moved from" in EPMC:
Leaf# egrep "moved from" epmc-trace.txt
2017 Sep 4 07:26:49.49690982:319903104:epmc_process_l3_upd:2299:t]IP 100.64.3.20 moved from MAC 5897.bde5.92ed to MAC 5897.bde5.92ec

2017 Sep 4 07:26:54.676146156:319917567:epmc_process_l3_upd:2299:t]IP 100.64.3.20 moved from MAC 5897.bde5.92ec to MAC 5897.bde5.92ed

2017 Sep 4 07:26:54.677959195:319917620:epmc_process_l3_upd:2299:t]IP 100.64.3.20 moved from MAC 5897.bde5.92ed to MAC 5897.bde5.92ec

2017 Sep 4 07:27:21.112399487:319971710:epmc_process_l3_upd:2299:t]IP 100.64.3.20 moved from MAC 5897.bde5.92ec to MAC 5897.bde5.92ed

2017 Sep 4 07:27:21.139985616:319971867:epmc_process_l3_upd:2299:t]IP 100.64.3.20 moved from MAC 5897.bde5.92ed to MAC 5897.bde5.92ed

The IP 100.64.3.20 is moving constantly between two MAC addresses

(only last octet in MAC is different)
Very likely a misconfigured server of VM port-group:
Using both uplink of a server without vPC/Port-channel on leaf
IP hash used instead of MAC pinning in Vmware…
Contracts
Traffic between EPGs is not allowed by Default
§ Every Bridge Domain can be segmented in
Security Zones called EPGs
§ Traffic within the EPG is allowed

§ Traffic between EPGs by default is not Bridge Domain

allowed

EPG 1 EPG 2 EPG 3

§ EPGs can be further segmented into

§ Micro EPGs based on: MAC, IP, VM tags

etc...
§ or Isolated EPGs
 VRFs can be configured not to allow EPG-to-EPG
communication, or to allow it, or to partially allow it

VRF1 VRF1 VRF1

Bridge Domain Bridge Domain Bridge Domain

EPG 1 EPG 2 EPG 3 EPG 1 EPG 2 EPG 1 EPG 2 EPG 3

EPG 3

Preferred Group
You can also configure EPGs for intra EPG
Isolation
• Intra EPG Isolation blocks communication
between all endpoints inside the group
• In the same EPG you can mix Physical and
Virtual endpoints
• This is similar to the concept of Private
VRF1
VLANs (Isolated VLANs)

Bridge Domain

EPG 1 EPG 2 EPG 3

 Intra EPG Isolation
Intra EPG Isolation
• Intra EPG Isolation blocks
communication between all endpoints
inside the group
• Supports mixing of Physical and Virtual
endpoints in same EPG
• Can be configured on all type of EPG

Intra EPG Isolation

<fvTenant name="Tenant1">
<fvAp name=”ap1">
<fvAEPg isAttrBasedEPg="no" matchT="AtleastOne" name="baseEPG" pcEnfPref=”enforced" prefGrMemb="exclude" prio="unspecified">
<fvRsBd tnFvBDName="bd"/>
</fvAEPg>
</fvAp>
</fvTenant>
Communication between EPGs is configured via
"contracts"

EPG 1 consumes "HTTP" EPG 2 provides "HTTP"

contract "HTTP"
EPG 1 EPG 2

ACL:
EPG 1 to EPG 2 dest port = 80
EPG 2 to EPG 1 source port = 80
What are Contracts in ACI
• They don’t steer traffic (except for service graph)
• They are ACLs
• They control redistribution of routes between VRFs
• Contracts are semantics to specify End Point Group (EPG) to EPG
communication in ACI Fabric
• Contracts can be between EPGs or between L3out and EPGs
• Filters take space in the Policy CAM
 Provider and Consumer is just to create a direction
between EPGs

EPG 1 consumes "EPG1-to-EPG2" EPG 2 provides "EPG1-to-EPG2"

contract "EPG1-to-EPG2"
EPG 1 EPG 2

Consumer-to-Provider direction

Provider-to-Consumer direction
You can then add subjects to the contract and
define the filters for each direction
CONSUMER PROVIDER

contract "EPG1-to-EPG2"
EPG 1 EPG 2

subject1: HTTP permit

subject2: Various other filters
subject3: etc..
vzAny

• One special EPG is called:

• vzAny or
• EPG collection for VRF or
VRF1
• AnyEPG

Bridge Domain

• A contract defined for vzAny

includes all the EPGs under the EPG 1 EPG 2 EPG 3

VRF and the L3Out also

A Common Use case for vzAny is for Shared
Services across VRFs
VRF SHARED SERVICES

Consumer
only 1 contract between
vzAny and EPG Provider
VRF1
Bridge Domain

EPG PROVIDER OF
SHARED SERVICES Provider Bridge Domain

EPG 1 EPG 2 EPG 3

Backup All VMs need to communicate with server BACKUP

There are multiple options to allow any-to-any
communication among EPGs
• Configuring the VRF in unenforced mode:
• This is a valid option, but it limits the possibility to use other features, for
instance service graph redirect
• Configuring a full mesh of EPGs that provide and consume the
same contract with permit
• Configuring vzAny to allow all traffic in the VRF
• Configuring Preferred Groups
 Contract Preferred Group
Inside the
Preferred Group there VRF – MyVRF
is unrestricted
communication Preferred Group

EPG-A EPG-B EPG-C EPG-D

L3Out
External
EPG

From between non-

preferred-groups EPGs
Excluded EPGs can and regular EPGs you
NOT communicate EPG-1 Contract-1
need contracts between
without contracts the individual EPGs
EPG-3

EPG-2
Contract-2
EPG Contract Inheritance

EPG_A Consumes Provides

Contract_DNS Contract_SSL
Contract_Internet

• Contract Inheritance simplifies the management of contracts

• Define an EPG that you want to use as a template
EPG_B is configured to inherit from EPG_A
EPG_A Consumes Provides

Contract_DNS Contract_SSL
Contract_Internet

EPG_B Consumes Provides

Contract_DNS Contract_SSL
Contract_Internet

Use the same contracts

as EPG_A
EPG_B is configured to inherit from EPG_A
You can now add more specific contracts
EPG_A Consumes Provides

Contract_DNS Contract_SSL
Contract_Internet

EPG_B Consumes Provides

Contract_DNS Contract_SSL
Contract_Internet Contract_TomCat

EPG_B also provides

another contract
The contract filters are programmed in the Policy Cam on the Leaf

49166 32787

leaf1# show zoning-rule scope VNID-OF-THE-VRF

How to Check in the TCAM
• log into the leaf of interest
• “show zoning-rule”
• You find “rule ID, and the SrcEPG and DestEPG
• The srcEPG and DestEPG are numbers called class-id
• You can find the class-id from the GUI by highlighting the EPG
Class-Ids of the EPGs
Contracts Seen from the EPG View
How to check if the contract that you added is
being hit and by which traffic (1)
• Add permit + log to the entries
that you want to verify
• Verify the flow table
• Verify the packet table and the
timestamps
How to check if the contract that you added is
being hit and by which traffic (2)
• Add permit + log to the entries that you want to verify
• Verify the flow table: Under Tenant/Operational/Flows, L3
Permit (if traffic is routed)
• Verify the packet table and the timestamps
How to check if the contract that you added is
being hit and by which traffic (3)
• Add permit + log to the
entries that you want to
verify
• Verify the flow table
• Verify the packet table
and the timestamps
Finding pcTag and scope in Object model

• fvAEPg is the class in logical model representing regular epg

• l3extInstP is the class in logical model representing L3 out EPG
admin@pod2-apic1:~> moquery -c fvAEPg | egrep -9 "^dn.*DC.*EPG" | egrep "dn|fv.AEP|scope|pcTag"
# fv.AEPg
dn : uni/tn-DC/ap-App/epg-EPG1
pcTag : 32770
scope : 2097153 For traffic between VM A and VM B we are interested
# fv.AEPg
dn : uni/tn-DC/ap-App/epg-EPG2 In rules between sclass 49153 and 32770 in scope 2097153
pcTag : 49153
scope : 2097153

admin@pod2-apic1:~> moquery -c l3extInstP | egrep -10 "^dn.*DC" | egrep "dn|l3ext|scope|pcTag"

# l3ext.InstP
dn : uni/tn-DC/out-L3out/instP-L3outEPG
pcTag : 32772
scope : 2097153
 Zoning in 4.x software
• Output of zoning rule was revamped in 4.0
• Policy_mgr stats still similar
leaf1# show zoning-rule scope 2097153 src-epg 32770
+---------+--------+--------+----------+---------+---------+---------+---------------+--------+---------------+
| Rule ID | SrcEPG | DstEPG | FilterID | Dir | operSt | Scope | Name | Action | Priority |
+---------+--------+--------+----------+---------+---------+---------+---------------+--------+---------------+
| 4142 | 32770 | 49153 | 39 | uni-dir | enabled | 3014656 | CT1-deny-test | deny | fully_qual(7) |
| 4145 | 32770 | 49153 | 10 | bi-dir | enabled | 3014656 | CT1-deny-test | permit | fully_qual(7) |
+---------+--------+--------+----------+---------+---------+---------+---------------+--------+---------------+
leaf1# show system internal policy-mgr stats 4140
Requested Rule Statistics
Rule (4140) DN (sys/actrl/scope-3014656/rule-3014656-s-16398-d-49157-f-39) Ingress: 0, Egress: 0, Pkts: 0
RevPkts: 0

leaf1# show system internal policy-mgr stats | egrep "4142|4145"

Rule (4142) DN (sys/actrl/scope-2097153/rule-2097153-s-32770-d-49153-f-5) Ingress: 533, Egress: 0
Rule (4145) DN (sys/actrl/scope-2097153/rule-2097153-s-49153-d-32770-f-5) Ingress: 0, Egress: 0
leaf1# show system internal policy-mgr stats | egrep "4142|4145"
Rule (4142) DN (sys/actrl/scope-2097153/rule-2097153-s-32770-d-49153-f-5) Ingress: 543, Egress: 0
Rule (4145) DN (sys/actrl/scope-2097153/rule-2097153-s-49153-d-32770-f-5) Ingress: 0, Egress: 0
 Checking Filter on the switch

Also we can check «show zoning-filter »

pod2-leaf1# show zoning-filter

FilterId Name EtherT ArpOpc Prot MatchOnlyFrag Stateful SFromPort SToPort DFromPort DToPort Prio Icmpv4T Icmpv6T TcpRules

======== ========== ====== ========= ======= ====== ======= ======= ==== ==== ==== ========= ======= ========

implicit implicit unspecified unspecified unspecified no no unspecified unspecified unspecified unspecified implicit unspecified unspecified

implarp implarp arp unspecified unspecified no no unspecified unspecified unspecified unspecified dport unspecified unspecified

default any unspecified unspecified unspecified no no unspecified unspecified unspecified unspecified def unspecified unspecified

5 5_0 ip unspecified icmp no no unspecified unspecified unspecified unspecified sport unspecified unspecified
Lab4:
Working with Contracts
Service Graph
Cisco ACI Service Insertion
Extending ACI Policy Model to L4-L7 Services

Application Centric Infrastructure Building Blocks

Physical + Virtual
Traditional
3-Tier FW
WEB ACC APP DB
Application ADC
APPLICATION
NETWORK PROFILE

CONTROLLER POLICY MODEL NEXUS 9300 AND 9500 L4-L7

Policy Model Extended to L4-L7

Building blocks of ACI

Application: 3 tier application (WEB-APP-DB) This may use ADC, FW services

Policy model: Define QOS, Security, Network, L4-L7 etc. to be applied to EPG
Managed vs. Unmanaged Mode

Network only switching feature adds the flexibility for customer to use only network
automation for service appliance. The configuration of the L4-L7 device is left to be done by
customer.
Customer can keep current L4-L7 device administration.

1: configure ACI Fabric for

L4-L7 service appliance

2: configure L4-L7 service appliance

L4-L7 Admin
 Why Service Graph Redirect:
Use different Firewall based on source FW1

Redirect
provider consumer
EPG-A Contract-A
L3Out

EPG-B Contract-B
provider consumer
Redirect
FW2

EPG-A goes to L3out via EPG-B goes to L3out via FW2

FW1

L3Out

EPG-A EPG-B
FW1 FW2
 Why Service Graph Redirect:
Inspect specific traffic by FW.

consumer provider

EPG EPG
Client
Contract Web

Redirect
Subject1: permit-http
Only HTTP traffic is redirected to FW, and Subject2: permit-all
then traffic is going to Web endpoint

Other traffic permitted by contract are

going to Web endpoint directly.

EPG Client EPG Web

How Service Graphs work The Device tells us how many
interfaces and logical
connectors on the Service
Devices
Service Graph Templates
defines HOW traffic should flow L1 L2

EP1 EP2
Redirect policies
In case of PBR defines
EPG EPG
Redirect next-hop info Client Web
Shadow
EPG

The Device Selection Policy

Contract selects traffic to defines how the Device will
redirect communicate with the fabric
 External and internal
What are shadow EPGs? interfaces
L1 L2
A ‘two armed’ example

• Shadow EPGs connect to the service

Device EP1 EP2
• External Interface is called the
“Consumer Connector” EPG Shadow EPG
• Internal interface is the “Provider Client EPG Web

Connector”
• Each is represented by a VLAN and Provider Connector
has its own PCTag

Cons Prov

Consumer Connector
Service Graphs Components
A quick review

• Devices
• Physical Device & interfaces it connects to in fabric. Converted to Consumer Connector and Provider
Connector
• A “Device” might be a cluster of multiple Appliance/VM

• Service Graph Template

• Define the flow of traffic

• Contract
• Places Contract between Consumer & Provider and the shadow EPG

• Device Selection Policy

• Ties the physical device to a Graph template and contract
Flooding and Multicast in
ACI Fabric
 FTAG Root for FTAG Root for FTAG Root for FTAG Root for
Trees 0,4,8 Trees 1,5,9 Trees 2,6,10 Trees 3,7,11

FTAG topologies are FTAG topologies are

advertised via IS-IS using limited to spine to leaf
martian addresses with links and only if a
the final 4 bits set to the downlink is lost will the
FTAG, 0.0.0.<ftag-id> FTAG topology pass
through a leaf

• To improve load balancing Multicast traffic is distributed across 12 FTAG topologies in the Fabric
• Destination groups are hashed across the FTAG topologies
• FTAG trees are rooted at spine switches (roots determined by APIC)
• FTAG tree calculation is performed by IS-IS and will create the FTAG trees as maximally redundant graphs
• An FTAG should give a non looped path reaching every leaf and spine
Useful mostly for MultiPod environments to check which Spine sent some traffic to IPN.
pod2-spine1# show isis internal mcast route ftag FTAG ID: 3 [Root] [Enabled] Cost:( 0/ 0/ 0)
IS-IS process: isis_infra ----------------------------------
VRF : default Root port: -
FTAG Routes OIF List:
==================================== Ethernet1/33.33
FTAG ID: 0 [Disabled] Ethernet1/34.34
Ethernet1/35.35
FTAG ID: 1 [Enabled] Cost:( 2/ 8/ 0) Ethernet1/36.36
----------------------------------
Root port: Ethernet1/34.34 FTAG ID: 4 [Root] [Enabled] Cost:( 0/ 0/ 0)
OIF List: ----------------------------------
Root port: -
FTAG ID: 2 [Root] [Enabled] Cost:( 0/ 0/ 0) OIF List:
---------------------------------- Ethernet1/33.33
Root port: - Ethernet1/34.34
OIF List: Ethernet1/35.35
Ethernet1/33.33
Ethernet1/36.36
Ethernet1/34.34
…
Ethernet1/35.35
Ethernet1/36.36
pod2-spine1# show isis internal mcast route
gipo § Usually to check the list of leaf switches that
IS-IS process: isis_infra
VRF : default have multicast destinations for a given BD
GIPo Routes § Leaf sends Group Membership LSPs (GM-
====================================
GIPo: 225.0.0.16 [TRANSIT] LSPs) for all its BDs with active multicast
OIF List: sources/receivers
Ethernet1/33.33
Ethernet1/34.34
Ethernet1/35.35
§ No GM-LSPs are triggered by Tenant
Ethernet1/36.36 multicast join/leave activity
GIPo: 225.0.3.160 [TRANSIT] § GM-LSPs are distributed throughout the
OIF List:
Ethernet1/34.34 fabric by ISIS using the same reliable flood
Ethernet1/35.35
mechanism used to distribute other LSPs
GIPo: 225.0.3.240 [TRANSIT]
OIF List: § GM-LSP from all nodes is used to build the
GIPo: 225.0.9.0 [TRANSIT] GIPo Distribution tree in the topology
OIF List:
Ethernet1/34.34
Ethernet1/35.35
§ Multicast address range pool must be a /15 for 128k addresses
§ From the infra default value you can determine what the pool address is.
§ For example, #.#.#.16 = #.#.#.0/15

apic01# moquery -c fvBD | grep -E "name|bcastP|dn" | grep -B 2 "infra"

name : default
bcastP : 225.2.0.16
dn : uni/tn-infra/BD-default
apic01#
ACI L2 Extension and STP
• L2 extension refers to extending a BD or an EPG to external devices
• There are essentially two ways to make this happen:
• Static binding to a path (port + encap-ID) inside an EPG
• This is exactly like adding a bare-metal host to an EPG
• Using an External Bridged Network configuration (EBN)
• Both approaches differ conceptually:
• Extending an EPG only extends that particular EPG
• Endpoint MAC addresses are learnt, BPDUs are forwarded inside the EPG
• L2out or External Bridged Networks (EBN) extend the entire BD
• EBN are governed by contracts (they are modeled as External EPGs)
• Endpoint MAC addresses are learned
• Contracts need to be configured between L2out and internal EPGs
EPG extension L2 out
• Only extend an EPG • Extend a full BD
• All EP behind the l2 • Contract can be configured
extended port are EP in the between L2 out and infra
EPG EPG
• All is part of same EPG
hence no contract can be
applied
• No STP running within ACI fabric
• BPDU frame is flooded within EPG. No
configuration required

DU

BP
BP

DU
• External switches break any potential
loop upon receiving the flooded BPDU
Same EPG
from ACI fabric BP
DU

• BDPU guard is enabled on FEX ports by

default
STP Root Switch
• User can enable BDPU guard on leaf
edge ports
• BPDU for PVST and RPVST has VLAN tagging and
will be flooded within the EPG for the VLAN. BPDU is
flooded when the EPG is configured for data traffic

DU

BP
BP

DU
• BPDU frame for MST is sent over the native VLAN and
doesn’t have VLAN tagging MST Region
Enable native BP
DU
• By default, ACI leaf port is not in VLAN 1 and VLAN 1 VLAN
is not native VLAN on leaf ports

• Different behavior than traditional switch STP Root Switch

• Leaf ports that connect to legacy network running
MST or third party switches that use standard (R)STP
• explicitly configure EPG for an untagged VLAN and
assign relevant ports to this EPG.
Otherwise BPDU frame won’t be flooded through data
forwarding works
 ACI Interaction with STP in Legacy Network
STP TCN Snooping

• Fabric intercept the BPDU TCN frame

• APIC flushes the MAC address for the corresponding
EPG that has the STP topology change
L4 L5 L6
• Bridge domain flooding vs. Convergence time with
L2 L3 Topology change
TCN. L1
results in new
blocked link.
• Can be seen in epmc-trace
• With MSTP user need to configure instance to VLAN sw1 sw2
mapping so APIC knows for what EPGs it need to flush
the MAC
• Recommend to have vPC connection to legacy Sw3 Root
switches to minimize the TCN
MacA
Example :
Mac A is behing Leaf2 , if we lose link between
Sw1 and Sw3, we need to unblock Sw2 to L5
à ACI can’t keep track of MAC A behind Leaf 2!
 ACI Interaction with MST in Legacy Network
TCN Snooping
• For PVST and RPVST, VLAN tagging in TCN frame indicates the VLAN that has topology, APIC flush the end points for
the corresponding EPG for the VLAN

• For MST, TCN frame indicates instances that has topology change. The instance to VLAN mapping has to be configured
on APIC in order for APIC to flush end points for the correct encap VLANs.
• EPG VLAN is configured consistently across leaf switches and neighboring
switches. BPDUs don’t get forwarded otherwise and this could lead to loop
• Pay attention to LLDP/CDP faults for native VLAN mismatch. They are
designated as critical faults since native VLAN mismatch can lead to loop

• MCP is a lightweight protocol designed to protect against loops that cannot

be discovered by either STP or LLDP
• MCP probe is sent, and if the fabric sees that the packet comes back in, then the fabric knows that
there is a loop and the fabric takes action based on that event

• You should enable MCP on all ports facing external switches or similar devices
 MCP CLI Sample Output
ACI-Leaf1#show mcp internal info interface ethernet 1/3 ACI-Leaf1#show mcp internal info vlan 1022
------------------------------------------ ---------------------------------------------
Interface: Ethernet1/3 PI VLAN: 36 Up
Native PI VLAN: 0 Encap VLAN: 1022
Native Encap VLAN: 0 PVRSTP TC Count: 28
BPDU Guard: disabled RSTP TC Count: 0
BPDU Filter: disabled Last TC flush at Fri Apr 21 14:14:00 2017
Port State: up on Ethernet1/3
Layer3 Port: false
Switching State: enabled
Mac Address: e0:e:da:a2:f1:e5
Interface MCP enabled: true
------------------- STP STATS --------------------
MSTP Count: 0
RSTP Count: 0
MSTP TC Count: 0
RSTP TC Count: 0
PVRSTP TC Count: 74
TCN Count: 0
PVID Error BPDU Count: 0
Error Packet Count: 0
BPDU Guard Event Count: 0
Last TC received at Fri Apr 21 14:14:02 2017
Lab5:
Integrate/Migrate L2
Network to ACI
L3Out Datapath
 Why L3OUT? • What is L3OUT for ?
Ø To connect ACI with other network domain
= devices with multiple subnet behind it
• How is L3OUT different from EPG?
Ø Speak Routing Protocol
VRF Ø No IP learning as endpoint
Ø Next-hop IP is stored in ARP table
= Same as normal routers
BD Subnet A L3OUT Routing Protocol
Next-hop MAC in endpoint table

EPG EPG L3OUT EPG leaf1# show endpoint vlan 84

84/TK:VRF1 vxlan-14876665 0000.0000.R1R1 L po3

EP A1 EP A2 EP A3 R1 Next-hop IP in ARP table (only for L3OUT)

MAC R1
leaf1# show ip arp vlan 84
IP R1 Address Age MAC Address Interface
R.R.R.1 00:07:51 0000.0000.R1R1 vlan84
MAC A1 MAC A2 MAC A3
IP A1 IP A2 IP A3
IP A4

Routes via Routing Protocol

leaf1# show ip route vrf TK:VRF1
X.0.0.0/8, ubest/mbest: 1/0
IP X.X.X.X/8 *via R.R.R.1, vlan84, [110/5], 2d00h, ospf-default, intra
IP Y.Y.Y.Y/8 Y.0.0.0/8, ubest/mbest: 1/0
*via R.R.R.1, vlan84, [110/5], 2d00h, ospf-default, intra
IP Z.Z.Z.Z/8 Z.0.0.0/8, ubest/mbest: 1/0
*via R.R.R.1, vlan84, [110/5], 2d00h, ospf-default, intra
 L3OUT Key Components
1. Learn external routes
1 Ø Routing Protocol in L3OUT
Distribute
External Routes 2. Distribute external routes to other
VRF Overlay-1 2 2
leaves
Ø MP-BGP
non-BLEAF BLEAF

3. Advertise internal routes (BD subnet)

VRF1 VRF1
to outside
3 3 Ø Redistribution
1
BD Subnet A L3OUT
and
EPG L3OUT EPG
Ø Contract
Advertise Learn
Internal Routes External Routes
(Export) (Import)
4. Allow traffic with contracts
4
4 Ø L3OUT EPG (Prefix Based
EPG)
 L3OUT Key Components
2. Distribute External Routes = MP-BGP in infra
1. Select ACI BGP AS and Route Reflector SPINEs 2. Apply Route Reflector policy to Pod Policy Group

Use default

3. Apply Pod Policy

※ L3OUT BGP share this same AS with the internal MP-BGP Group to Pod Profile
 CLI Verification
1. Do both border leaf and non-border leaf have BGP sessions with RR spines?
leaf# show bgp sessions vrf overlay-1
Neighbor ASN Flaps LastUpDn|LastRead|LastWrit St Port(L/R) Notif(S/R)
10.0.184.65 65003 0 2d07h |never |never E 37850/179 0/0
10.0.184.66 65003 0 2d07h |never |never E 45089/179 0/0

leaf# acidiag fnvread | grep spine

1001 1 spine1 FGE10000000 10.0.184.65/32 spine active 0
1002 1 spine2 SAL10000000 10.0.184.66/32 spine active 0

2. Is the external route learned on a border leaf?

border-leaf# show ip route vrf TK:VRF1
10.0.0.0/8, ubest/mbest: 1/0
*via 15.0.0.1, Vlan58, [110/5], 2d08h, ospf-default, intra

3. Does non-border leaf show the expected border leaf as next-hop?

non-border-leaf# show ip route vrf TK:VRF1
10.0.0.0/8, ubest/mbest: 2/0
*via 10.0.184.67%overlay-1, [200/5], 2d08h, bgp-65003, internal, tag 65003
*via 10.0.184.64%overlay-1, [200/5], 2d08h, bgp-65003, internal, tag 65003

non-border-leaf# acidiag fnvread

ID Pod ID Name Serial Number IP Address Role State LastUpdMsgId
--------------------------------------------------------------------------------------------------------------
103 1 leaf3 SAL10000003 10.0.184.64/32 leaf active 0
104 1 leaf4 SAL10000004 10.0.184.67/32 leaf active 0
• Three main areas for config:
• L3Outside (L3ext):
• OSPF, EIGRP, BGP, Static
• Routing protocol options
• Logical Node Profile:
• Nodes/Interface definition for the
External connections
• External Network Instance (EPG)
• Subnets to be imported/exported
Config Steps
1. Configure
1. L3 out global (VRF, Phys Dome, which RP to use)
2. L3 out node (BL) including RID (+ loopback) + if BGP the peering
3. Configure Logical interface aka if on BL to peer
Note external prefix will be seen in fabric automatically by default
2. Configure one or more L3 out EPG
1. Includes subnet in it (or use default 0.0.0.0/0) à assignement of external
network to the EPG
2. Configure Contract
3. Decides what to advertise out
1. BD subnet
2. Or other network from other L3 out (transit)
 L3OUT Key Components
1. Learn External Routes = Routing Protocol
Configurations

External Routed Networks (L3OUT)

• VRF to deploy Routing Protocol Routing Protocol

• Routing Protocol parameters Information
ex. OSPF area 0.0.0.1 nssa

Only on configured
Node Profile nodes
(Border LEAF)
• Node(s) to deploy Routing Protocol
• Static Route (if any)

Interface Profile

• I/F(s) to deploy Routing Protocol

• Routing Protocol I/F parameters VRF1 VRF1 VRF1
ex. OSPF hello interval
BD L3OUT L3OUT
Networks (L3OUT EPG)

• Contract ※ Details for each protocol are in later sections

• Advanced Route Control ※ Details for L3OUT EPG are in later sections
ex. route-map
L3 out Wizard (Revamped in 4.2) – Step 1
L3 out Wizard (Revamped in 4.2) – Step 2
Select your type of
L3 interface and l2 path

You can add more

Node (BL)
And/or more If per BL
Step 3 – protocol policies
L3 out Wizard (Revamped in 4.2) – Step 4

If you unclick use

Default EPG for ext
Network. You will be
Able configure subnet
One by one
• Subnet classification in L3Out
(assign subnet to EPG name)
• Flags are optional
1. Export Route Control Subnet
• Only used in transit routing
• Use to advertise transit subnet out
• Advertise prefix received from other L3Out (or GOLF) to
L3Out EPG where this flag is used.
• Adds entry in prefix-list of the outbound route-map.
• Should NOT be used on subnets/prefixes:
• Bridge Domain (internal) prefixes.
• Prefixes that are received by that L3Out.
• Does not rely on received prefixes (can be a fake subnet).
• Only modifies outgoing route-map; does not add route (see above)
• Route-map is on BGP neighbor or OSPF area-filter.
leaf2# show ip ospf vrf DC:DC | egrep filter
Area-filter in 'exp-ctx-proto-3014656‚
leaf2# show route-map exp-ctx-proto-3014656
route-map exp-ctx-proto-3014656, permit, sequence 7801
Match clauses:
ip address prefix-lists: IPv4-proto32772-3014656-exc-ext-inferred-export-dst
ipv6 address prefix-lists: IPv6-deny-all
Set clauses:
tag 4294967295
leaf2# show ip prefix-list IPv4-proto32772-3014656-exc-ext-inferred-export-dst
ip prefix-list IPv4-proto32772-3014656-exc-ext-inferred-export-dst: 1 entries
seq 1 permit 1.1.0.0/16
• Disabled by default (greyed out)
• Only enabled if the L3Out is set with import
route control enforcement (BGP or OSPF)
• Works the same as Export Route Control
Subnet but on the import route-map

leaf2# show ip bgp neigh vrf RD:SB | egrep route-map

Outbound route-map configured is exp-l3out-L3out-peer-2097153, handle
obtained
Inbound route-map configured is imp-l3out-L3out-peer-2097153, handle obtained

leaf2# show route-map imp-l3out-L3out-peer-2097153

route-map imp-l3out-L3out-peer-2097153, permit, sequence 7801
Match clauses:
ip address prefix-lists: IPv4-peer49153-2097153-exc-ext-inferred-import-dst
ipv6 address prefix-lists: IPv6-deny-all
Set clauses:

leaf2# show ip prefix-list IPv4-peer49153-2097153-exc-ext-inferred-import-dst

ip prefix-list IPv4-peer49153-2097153-exc-ext-inferred-import-dst: 1 entries
seq 1 permit 2.2.0.0/16
3. External Subnet for External EPG
• By far the most important!
• To assign external prefix to the correct L3Out EPG
• Used both for ingress traffic and egress traffic
• Typically contains external prefix(es) expected by
that L3Out in incoming direction
• SHOULD NOT contain:
• Bridge Domain (internal) prefixes
• Prefixes that are received by other L3Out EPGs

• Might contain default internet route 0.0.0.0/0

• only on one L3Out per VRF
• Important: EPG assignment is based on LPM
lookup
• Transit prefix
• To export (receiving subnet on L3Out1 and transmitting subnet on L3Out2)
• Use only: Export Route Control Subnet on L3Out2.

• Received prefixes on a layer 3 out.

• Use External Subnet for External EPG.

• Normally there is never a need to mix both flags.

• 0.0.0.0/0 with External Subnet for External EPG.
• Only on single L3Out per VRF (the one facing external world/internet).
• Other L3Out
• More specific subnet use only External Subnet for External EPG.
§ First L3Out with 0.0.0.0/0 as Ext Sub for Ext EPG.
§ EPG has an ICMP contract filter with an internal EPG with pcTag 32774.
§ L3Out EPG has pcTag 5480, not used due to 0/0.
§ Adding Second L3Out with 0.0.0.0/0 as Ext Sub for Ext EPG sub.
§ EPG has an HTTP contract filter with an internal EPG with pcTag 49156.
§ Aclqos prefix remains the same (15 for default route), but two different tcam entry.
pod2-leaf2# show zoning-rule | egrep "32774.*5280"
Rule ID SrcEPG DstEPG FilterID operSt Scope Action Priority
======= ====== ====== ======== ====== ===== ====== ========
4209 32774 15 5 enabled 2785280 permit fully_qual(6)
4212 32774 15 21 enabled 2785280 permit fully_qual(6)

pod2-leaf2# show zoning-filter

5 5_0 ip unspecified icmp no no unspecified unspecified unspecified unspecified 22
21 21_0 ip unspecified tcp no yes unspecified unspecified http http

Conclusion: Packet entering from EPG 32774 can be either ICMP or HTTP.
Drawback: Packet entering EPG 32774 with ICMP or HTTP can go out L3Out1 or L3Out2, if the
route to destination is reachable.
Advertise BD under subnet
EIGRP basic check
bdsol-aci32-leaf1# show ip eigrp vrf RD-BGP:RD
IP-EIGRP AS 1 ID 172.16.1.1 VRF RD-BGP:RD
Process-tag: default
Instance Number: 1
Status: running
Authentication mode: none Check
Authentication key-chain: none Eigrp process is running
Metric weights: K1=1 K2=0 K3=1 K4=0 K5=0 And has some interface running eigrp
metric version: 32bit
IP proto: 88 Multicast group: 224.0.0.10
Int distance: 90 Ext distance: 170 Note as well all Route-map used
Max paths: 8 To redistribute to eigrp
Active Interval: 3 minute(s)
Number of EIGRP interfaces: 2 (1 loopbacks)
Number of EIGRP passive interfaces: 0
Number of EIGRP peers: 1
Redistributing:
static route-map exp-ctx-st-2654211
ospf-default route-map exp-ctx-proto-2654211
direct route-map exp-ctx-st-2654211
coop route-map exp-ctx-st-2654211
bgp-132 route-map exp-ctx-proto-2654211
Tablemap: route-map exp-ctx-2654211-deny-external-tag , filter-configured
Graceful-Restart: Enabled
Stub-Routing: Disabled
NSF converge time limit/expiries: 120/0
NSF route-hold time limit/expiries: 240/0
EIGRP interface check
bdsol-aci32-leaf1# show ip eigrp interface vrf RD-BGP:RD
IP-EIGRP interfaces for process 1 VRF RD-BGP:RD

Xmit Queue Mean Pacing Time Multicast Pending

Interface Peers Un/Reliable SRTT Un/Reliable Flow Timer Routes
Lo11 0 0/0 0 0/0 0 0
Hello interval is 5 sec
Holdtime interval is 15 sec
Next xmit serial <none>
Un/reliable mcasts: 0/0 Un/reliable ucasts: 0/0
Mcast exceptions: 0 CR packets: 0 ACKs suppressed: 0
Retransmissions sent: 0 Out-of-sequence rcvd: 0
Authentication mode is not set
Use multicast
Classic/wide metric peers: 0/0
Vlan129 1 0/0 1 0/0 50 0
Hello interval is 5 sec
Holdtime interval is 15 sec
Check
Next xmit serial <none>
Un/reliable mcasts: 0/3 Un/reliable ucasts: 2/2 The interface runs eigrp
Mcast exceptions: 0 CR packets: 0 ACKs suppressed: 1 (at least the l3 out logical interface and
Retransmissions sent: 0 Out-of-sequence rcvd: 0 the loopback)
Authentication mode is not set
Use multicast
Classic/wide metric peers: 1/0
Checking EIGRP neighbor
bdsol-aci32-leaf1# show ip eigrp neigh det vrf RD-BGP:RD
IP-EIGRP neighbors for process 1 VRF RD-BGP:RD
H Address Interface Hold Uptime SRTT RTO Q Seq
(sec) (ms) Cnt Num
0 172.16.1.82 Vlan129 12 00:26:44 1 50 0 3
Version 8.0/1.2, Retrans: 0, Retries: 0, BFD state: N/A, Prefixes: 1

Check if we have an EIGRP neighbor

EIGRP sending BD subnet
bdsol-aci32-leaf1# show ip eigrp vrf RD-BGP:RD | egrep route-ma
static route-map exp-ctx-st-2654211
ospf-default route-map exp-ctx-proto-2654211
direct route-map exp-ctx-st-2654211
coop route-map exp-ctx-st-2654211
bgp-132 route-map exp-ctx-proto-2654211
Tablemap: route-map exp-ctx-2654211-deny-external-tag , filter-configured
bdsol-aci32-leaf1# show route-map exp-ctx-st-2654211
..
route-map exp-ctx-st-2654211, permit, sequence 15804
Match clauses:
ip address prefix-lists: IPv4-st16387-2654211-exc-int-inferred-export-dst
ipv6 address prefix-lists: IPv6-deny-all
Set clauses:
tag 0
..
bdsol-aci32-leaf1# show ip prefix-list IPv4-st16387-2654211-exc-int-inferred-export-dst
ip prefix-list IPv4-st16387-2654211-exc-int-inferred-export-dst: 1 entries
seq 1 permit 172.16.10.0/24
EIGRP BD subnet in EIGRP topology DB
bdsol-aci32-leaf1# show ip eigrp topology 172.16.10.0/24 vrf RD-BGP:RD

IP-EIGRP (AS 1): Topology entry for 172.16.10.0/24

State is Passive, Query origin flag is 1, 1 Successor(s), FD is 51200
Routing Descriptor Blocks:
0.0.0.0, from Rconnected, Send flag is 0x0
Composite metric is (51200/0), Route is External
Vector metric:
Minimum bandwidth is 100000 Kbit
Total delay is 1000 microseconds
Reliability is 255/255
Load is 1/255
Minimum MTU is 1492
Hop count is 0
Internal tag is 0
External data:
Originating router is 172.16.1.1 (this system)
AS number of route is 0
External protocol is Static, external metric is 0
Administrator tag is 0 (0x00000000)
• Leaf components:
• DataPlane ASIC
(Gen-1, Gen-2) Control plane
• Control Plane Linux
Linux Mgmt phy
kpm_inb eth0 interface
• eth0 for OoB
• Connection to Mgmt interface ARP, CDP,
Data plane LLDP, LACP,
• kpm_inb ASIC BGP, ...

• control protocols
Leaf switch
• good place for troubleshooting
pod2-leaf1# netstat -tupn | egrep 179 Is TCP session established for BGP?
(No info could be read for "-p": geteuid()=15374 but you should be root.)
tcp 0 0 10.33.10.1:179 10.33.10.2:62415 ESTABLISHED -
tcp 0 0 10.0.168.95:44599 10.0.168.94:179 ESTABLISHED -
tcp 0 0 10.0.168.95:57261 10.0.168.92:179 ESTABLISHED -

Are BGP packets received / transmitted?

pod2-leaf1# tcpdump -i kpm_inb -f port 179
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on kpm_inb, link-type EN10MB (Ethernet), capture size 65535 bytes
22:38:59.188850 IP 10.33.10.2.62415 > 10.33.10.1.179: Flags [P.], seq
2891078262:2891078281, ack 1176057492, win 17376, options [nop,nop,TS val 172217473 ecr
506104871], length 19: BGP, length: 19
22:38:59.188921 IP 10.33.10.1.179 > 10.33.10.2.62415: Flags [.], ack 19, win 145, options
[nop,nop,TS val 506113132 ecr 172217473], length 0
§ Typically ACI does not store ARP info in show ip arp.
§ However, L2-attached IP on layer 3 (VLAN based) is an exception.
pod2-leaf1# show ip arp detail vrf L3:L3

Flags: * - Adjacencies learnt on non-active FHRP router

+ - Adjacencies synced via CFSoE
# - Adjacencies Throttled for Glean
D - Static Adjacencies attached to down interface

IP ARP Table for all contexts

Total number of entries: 1
Address Age MAC Address Interface Physical Interface
10.33.10.2 00:08:58 002a.6ab1.917c vlan65 eth1/33
Step 1: TCP dump on int kpm_inb to see if ARP packets are received
(Note: only Rx ARP are seen on this interface, not Tx) For ARP specifically, tcpdump on kpm_inb
only shows Rx packets.
pod2-leaf1# tcpdump -i kpm_inb arp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on kpm_inb, link-type EN10MB (Ethernet), capture size 65535 bytes
23:24:46.926593 ARP, Reply 10.33.10.2 is-at 00:2a:6a:b1:91:7c (oui Unknown), length 42
Log details: (bottom to top)
§ Received arp from 10.33.10.2.
§ Added the arp entry.
Step 2: Check the ARP process debug log § Populated MIT object store with the arp info.
pod2-leaf1# show ip arp internal event-history event | egrep -B 1 "10.33.10.2" | more
156) Event:E_DEBUG_DSF, length:420, at 927332 usecs after Mon Apr 11 23:24:46 2017
[116] TID 9422:arp_objst_dynadj_handle_pri_chunk:1279: Object store handling: my_dn - sys/arp/inst/dom-L3:L3/db-ip/adj-
[vlan65]-[10.33.10.2], oper state - 2, arp_cfg_opc - 1, arp_mac - 002a.6ab1.917c, arp_ip
- 10.33.10.2, arp_ifindex - 0, arp_iod - 17, arp_iod - Vlan65, arp_phy_ifindex - 0,arp_phy_iod - 88, arp_phy_iod -
Ethernet1/33, context_id - 14, uptime(sec) - 6737, uptime(nsec) - 702545344, upTS - 1460417087363
--
157) Event:E_DEBUG_DSF, length:212, at 927263 usecs after Mon Apr 11 23:24:46 2017
[116] TID 9422:arp_add_adj:3898: Entry added for 10.33.10.2, 002a.6ab1.917c, state 2 on interface Vlan65, physical
interface Ethernet1/33, ismct 0. Rearp (interval: 0, count: 0), TTL: 1500 seconds, sent_to_am
: No
--
160) Event:E_DEBUG_DSF, length:134, at 927071 usecs after Mon Apr 11 23:24:46 2017
[116] TID 9422:arp_process_receive_packet_msg:6681: Check if garp. iod Vlan65 ifmode 128 src 10.33.10.2 dst 10.33.10.1
adj 0x97c0f594
--
Lab6:
Create or Integrate a
Layer 3 Connection in ACI
 Parameter Value
Admin >> Import/Export
Hostname/IP store1.demo.local
Remote Locations remlocBKP Protocol SFTP

Export Policies
Remote path /home/cfg-bkp
Remote port 22
Configuration Username cfgbkpadmin
Password ********
ACI-BACKUP

Parameter Value
Format JSON
Snapshot No
Export Destination remlocBKP Very important
Scheduler schedDaily to enable
Secure Props Enabled
• Reloading a switch is a fix, not a solution
• Sometimes a simple switch reload clears problems
• When switch reloads its configuration, it is cleared and freshly reconfigured by
APIC
• During power-cycle some interfaces might restart while their policy is not yet
programmed, causing problems
• Reload switch only if instructed by TAC or being aware of the
consequences
• It's very hard for TAC to troubleshoot a problem if the switch was reloaded
• On entering maintenance mode, the system
automatically shuts down all supported
protocols. In addition, all the front-panel
interfaces are shutdown on the switch
• Navigate to Fabric > Inventory > PodX and
right click the switch you wish to put in
maintenance mode
• Once in maintenance mode, the switch can
be reloaded and then returned to production
in a controlled manner
• On observing the faulty switch in the Fabric, the customer wants to isolate it
from the network so that traffic can flow through the rest of the fabric
• Customer wants to keep the isolated node for later debugging by support
engineer or Cisco TAC

?
• IS-IS set overload mode
• OSPF, EIGRP, and BGP do graceful shutdown
• vPC shuts down Keep-Alive & Peer Link
• Shutdown all front panel ports and directly
connected IFC ports
(Cuts Laser on the Port)

Leaf interface status remains up but it’s actually down on remote side. It’s
because we try to retain the information for further debugging.
• On active ACI fabrics TechSupport
can be several GB in size, making
remote storage a better choice
• Generating TechSupport will take
significant time to complete
• Remote location is defined by
navigating to Admin >
Import/Export > Remote
Locations > Actions > Create
Remote Location
• Once remote location is defined,
scheduled or on-demand,
TechSupport export can be
configured
• For on-demand TechSupport
individual switches and/or
controllers can be included in
operation
• Navigate to Admin >
Import/Export > TechSupport or
On-demand TechSupport
• Sometimes, while waiting for TechSupport, TAC can start support by
examining XML files exported from ACI
• End-result of XML export are smaller files that are generated faster
• Result files don't contain all the information that is collected using
TechSupport
• XML export is performed using icurl command on APIC
• Different UNIX tools can be used to modify output
• Result files then have to be manually transferred to external storage
Favorite XML for TAC - audit log
• What, when and who made changes on ACI fabric

APIC01# icurl 'http://localhost:7777/api/class/aaaModLR.xml' > aaaModLR.xml 2>

aaaModLR.err
APIC01# ls -l
total 56
-rw-r--r-- 1 admin admin 65 Sep 18 13:21 aaaModLR.err
-rw-r--r-- 1 admin admin 123223152 Sep 18 13:21 aaaModLR.xml
lrwxrwxrwx 1 root root 12 Oct 9 2015 aci -> /.aci/viewfs
lrwxrwxrwx 1 root root 13 Oct 9 2015 debug -> /.aci/debugfs
-rw-r--r-- 1 admin admin 47364 Jun 7 08:01 endpoint.txt
lrwxrwxrwx 1 root root 11 Oct 9 2015 mit -> /.aci/mitfs
catsAPIC01#
• Each leaf has an Endpoint Manager (EPM), which is a process that
tracks all of the endpoints on a particular device
• Collects all L2/L3 changes of end-points
• History horizon is sometimes very short, might be just several minutes
• If you detect failure happening:
• Store current EPM trace file, if possible on all leafs in data path
• Record time/date of failure
• Provide those information to TAC

• Then start collecting TechSupport

 • To collect EPM-trace files login to leaf
• Copy trace files to another location to avoid roll-over
LEAF101# cd /var/log/dme/log
LEAF101# ls epm* -l
-rw-rw-rw- 1 root root 47591424 Sep 18 14:12 epm-trace.txt
-rw-rw-rw- 1 root root 39321600 Sep 18 14:12 epmc-trace.txt
LEAF101# cp epm-trace.txt /tmp/$(date+"%y%m%d")-epm-trace.txt
LEAF101# cp epmc-trace.txt /tmp/$(date+"%y%m%d")-epmc-trace.txt
LEAF101# ls -l /tmp/20170918*
-rw------- 1 admin admin 47869952 Sep 18 14:14 /tmp/20200918-epm-trace.txt
-rw------- 1 admin admin 39444480 Sep 18 14:14 /tmp/20200918-epmc-trace.txt
LEAF101#
Lab7:
Immediate Reactive
Recovery with ACI
ACI Depoyment Models
ACI Anywhere

• Operational Simplicity: Same Container

s
Hypervisor

“look and feel” as On-Premise

• Automated Policy Translation:
ACI Anywhere
Consistency across the entire
data center
Cloud
• Common Governance: Exchange
Data
End-to-end discovery, visibility Center
and troubleshooting

On Premises
Cloud
IOT Edge

© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Cisco ACI Anywhere
Any Workload, Any Location, Any Cloud
ACI Anywhere
Remote Leaf / Virtual PoD APIC / Multi-Site Multi-Cloud Extensions

IP IP
WAN WAN

Remote Location On Premise Public Cloud

Security Everywhere Analytics Everywhere Policy Everywhere

 Single Fabric/Pod Topology Options
ACI mini Standard (2-Tier) Multi-Tier (3-Tier)

physical virtual

• From release 4.0 • From the first release (1.0) • From release 4.1
• For small deployment • Most popular and standard ACI • For DCs with cabling restrictions
• 1 APIC + 2 virtual APICs topology
• Up to 2 spine + 4 leaf switches
• Replacing virtual APICs with
physical APICs Not sure what to pick? Go with 2-Tier!
• Expand to a full ACI fabric More about ACI Mini:
https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/kb/Cisco-Mini-ACI-Fabric-and-Virtual-APICs.html
ACI Three-Tier Architecture
Migration from N2/5/7K to ACI

Two-Tier Two-Tier Three-Tier Replace FEX with 2nd Tier

Traditional ACI ACI N9300 Leaf
Better Visibility and Policy
Enforcement
Investment Protection: Reuse
Existing Cable Plan
Simplify N2/N5/N7k Migration
N2/5/7K N2/9/9K N9/9/9K to ACI

No hassle migration from Legacy

with Three-Tier ACI Architecture
Feature Evolution – Stretched Fabric

Spine Spine Spine Spine

Single
Fabric

Leaf Leaf Leaf Leaf Leaf Leaf

APIC APIC APIC

Feature Evolution – Stretched Fabric
• Transit leafs connect to all spines
• COOP, ISIS, and BGP extended across locations

Not scalable
Feature Evolution
What if ACI must be extended to other locations?

• TEP reachability must be communicated

• Endpoints must be synced across locations
• Mechanism needed for BUM traffic
• APIC Cluster must be extended

1
 Feature Evolution – Multipod
IPN

MP OSPF,PIM OSPF
F , IG , IGM • Single Fabric Extended
OSP P
BGP • Each pod is local
Spine Spine VPNv4/EVPN Spine Spine instance of ISIS and
ISIS ISIS COOP
Pod1 Pod2
COOP COOP
• Inter-pod connectivity
MPBG MPBG
P
through IPN
P
• Inter-pod BUM uses
Leaf Leaf Leaf Leaf Leaf Leaf PIM-Bidir
• BGP between pods to
share endpoints and
external routes
APIC APIC APIC

116
IPN Requirements
qOSPF

qDHCP relay
qJumbo MTU (9150 Bytes)
qRouted Subinterfaces
qPIM Bidir with at least /15 Mask
qQoS (optional)

117
 ACI Multi-Pod
Supported Topologies
Intra-DC Two DC sites directly connected

POD 1 POD n POD 1 Dark fiber/DWDM POD 2

APIC Cluster APIC Cluster

3 (or more) DC Sites directly connected Multiple sites interconnected by a

generic L3 network
POD 1 POD 2
Dark fiber/DWDM

L3
10G*/40G/100G

(up to 50 msec RTT) POD 3

 ACI Multi-Site
Multi-Site Orchestrator Site N
(MSO)
3 VM Cluster

VM VM VM VM VM VM VM

Any Routed IP Network

Site1 Site 2

VM VM VM VM VM VM VM VM VM VM VM VM VM VM

No Multicast <= 1s RTT Required (MSO APIC) Single central management (MSO)
Phased Changes (Zones) Up to 12 Sites, distributed gateway
ACI Multi-Site
Software and Hardware Requirements

• Support all ACI leaf switches (1st Gen, EX, FX, FX2) Any Routed Network

• Modular Spine with EX/FX line card to

connect to the inter-site network
Can have only a subset
1st Gen 1st Gen -EX -EX of spines connecting to
• Nexus 9364c or Nexus 9332x fixed spine the IP network
supported for Multi-Site from ACI 3.1
release
• 1st generation spines (including
Nexus 9336PQ) not supported
• Can still leverage 1st gen spines for intra-site leaf
to leaf communication
ACI Multisite Orchestrator –
Enables Distributed Data Centers
• Three MSO nodes are clustered and run
concurrently (active/active)
• Typical database redundancy considerations
REST (minority/majority rules)
GUI
API
• Up to 150 msec RTT latency supported
between MSO nodes
ACI Multi-Site Orchestrator Cluster
• OOB Mgmt connectivity to the APIC
MSO Node 1
150 msec RTT
(max) MSO Node 2 MSO Node 3 clusters deployed in separate sites
• Up to 1 sec RTT latency between MSO and
1 sec RTT
APIC nodes
(max)
IP network • Main functions offered by MSO:
• Monitoring the health-state of the different ACI
Sites
…..
Site 1 Site 2 Site n • Provisioning of day-0 infrastructure
configuration to establish inter-site EVPN
control plane and VXLAN data plane
• Defining and provisioning tenant policies
• Day-2 operation functionalities
ACI Edge Deployment Options

Remote Leaf
• Satellite/Remote locations deployment
• Leverage Nexus 9K hardware capabilities
Any Routed IP Network
• Extend ACI policy

ACI Virtual Pod

• Extend ACI Policy w/o H/W
• Virtualized Spine, Leaf and APIC
• Bare-metal providers, co-location
providers, legacy networks

ACI Mini Fabric

• Small scale and cost optimized
deployments
• 5RU: Ideal for space, power
and cooling restrictions
ACI On-Premise • Telco DC, small to midsize business
ACI Remote Leaf
Business Value and Use Cases

Extending the ACI policy model outside the main datacenter to remote
sites distributed over IP Backbone (Telco DCs, CoLo locations, etc.)

Extending ACI fabric policy and L2/L3 connectivity to a small DC site

without requiring the deployment of a full-blown ACI Fabric or for
migration/coexistence with legacy DC sites

Centralized Policy Management and Control Plane for remote locations

Small form factor solution at locations with space constraints

Architecture Overview
Remote Location contains Nexus 9300
connected to IP Network and fully
managed by APIC cluster of Main DC

APIC and Spine Nodes IP Network L2 / L3

remain at Main DC

• No Multicast
• HREP for BUM traffic
vSwitch `
Hypervisor

Bare
ACI Main DC Metal

Remote Location
Local Traffic forwarding between endpoints
ACI Remote Leaf Requirements
Hardware & Software

ACI Main DC Remote Location

Supported Spines Supported Leaf
• N93180YC-EX
Fixed Spine • N93108TC-EX
• N9364C • N93180LC-EX
• N9332C • N93180YC-FX
Modular Spine Line Cards • N93108TC-FX
• N9732C-EX • N9348GC-FXP
• N9736C-FX • N9336C-FX2

All hardware from –EX onwards is supported

 Data Center A
ACI vPod Use Cases

IP Network

Bare Metal Cloud

Data Center B

VM VM VM VM

Brownfield
Data Center C

VM VM VM VM VM VM VM

ACI Main Data Center

Co-location/Remote DC
ACI Virtual Pod (vPod)
Architectural Components

Management Cluster (vSpine + vLeaf)

• vSpine nodes: centralized endpoint and LPM database
Virtual Pod (COOP and BGP)

• vLeaf nodes: distribute APIC policies to ACI Virtual

Edges (DME/PE on vLeaf <-> Opflex on AVE)
vSpine vSpine
• vSpine and vLeaf nodes are not used for data-plane
Control Plane
forwarding

ACI Virtual Edge (vPod Mode)

vLeaf vLeaf • Implements ACI data plane function (switching and
routing) and policy enforcement
ACI Virtual Edge Data Plane • iVXLAN for communication within vPod and across
Pods
ACI vPod Requirements
Hardware & Software

On-Premises Data Center vPoD Data Center

Supported Spines • VMware vCenter running 6.0 or later
Fixed Spine • 2 hosts for Management cluster
• N9364C
• N9332C • Management cluster may exist on the
same AVE ESXi nodes
Modular Spine (C9504/C9508/C9516)
• N9732C-EX with N9K-C950x-FM-E(2) • ESXi 6.0 or 6.5
• Each vSpine (x2) & vLeaf(x2) VM consumes 4vCPU,
• N9736C-FX with N9K-C950x-FM-E(2) 16 GB RAM and 80 GB storage
APIC Controller Software • Each AVE (one per ESXi host) VM consumes 2vCPU,
8 GB RAM and 8 GB storage
• ACI 4.0+ onward release
 ACI Virtual Pod
Extend ACI to Brownfield or Baremetal Cloud Locations

Legacy DC Site / Baremetal Cloud

IP WAN

Layer 2
Layer 2
AVE AVE AVE
Hypervisor Hypervisor Hypervisor

ACI “Virtual Leaf” Nodes

Switching/routing and
policy enforcement • vPod allows to extend ACI connectivity and policies to compute resources deployed in
legacy DC networks
• No need to deploy any ACI HW in the remote network
Cloud ACI – Multicloud Extensions
Cloud Service Connectivity
Public Cloud
Public Cloud Public Cloud Bare Metal Cloud B

Container Hypervisor
s
ACI Anywhere ACI Virtual
ACI
Data Center ACI Anywhere ACI
Anywhere

Internet
Compute Edge
(Branch)
MPLS

Cloud
On Premises Exchange
Cloud

Automation Security Mobility Visibility

 February 2019

Simplified Multi-Cloud Connectivity ACI 4.1

Benefits

1 Single point for orchestration and configuration

Automated connectivity without having to provisioning complex

2
routing protocols

3 Scale within and across data centers and public clouds

4 Reduced overhead through integration of underlay and overlay

5 End to end visibility, monitoring and troubleshooting

6 Automated life cycle management of Infra

Extending ACI to the Cloud
ACI Multi-Site
Orchestrator

ACI for On-Premise

SG SG SG
SG Rule SG Rule
Web APP DB
IP
Network
AWS Region

Cloud ACI for Public Cloud

IP
Network
EPG EPG EPG
Contract Contract
Web VM VM APP
VM DB

ASG ASG ASG

NSG NSG
Web APP DB
Azure Region
 February 2019

Cloud ACI - Extensions to AWS ACI 4.1

Simplified Multi-Cloud Connectivity

On-Prem DC Public Cloud DC

ACI MSO

VXLAN (Data Plane)

CSR1kv
BGP EVPN (Control Plane)

Amazon Gateway Amazon Gateway

User VPC 1 User VPC 2
IPSec Tunnel AWS Instances AWS Instances
(Underlay)

Automated network Encrypted AWS Multi-region End-to-end control plane for

extension to AWS tunnel option Support route and policy exchange
 February 2019

Cloud ACI - Extensions to AWS ACI 4.1

Consistent Security Posture

On-Prem DC Public Cloud DC

ACI MSO

EPG EPG
Contract Contract EPG DB SG Web SG Rule SG App SG Rule SG DB
Web App

ACI MSO as single Lifecycle management

Automated policy translation to of CSR1000v and Consistent Policy
point of policy management and
native AWS constructs AWS VGW Extensions across sites
orchestration
 February 2019

Cloud ACI - Extensions to AWS ACI 4.1

Service Automation for Multi-Cloud

On-Prem DC Public Cloud DC

ACI MSO

Load balancer Firewall AWS Elastic AWS Firewall

Load Balancer

Capabilities for native cloud Ability to do services Service chain AWS Native
services (native cloud load stitching from cloud to a services (Elastic Load Balancer)
balancers, target groups) L4-L7 on-premises based on policies
 February 2019

Service Automation for Multi-Cloud ACI 4.1

Benefits

1 Service chaining for Cloud Workloads*

2 Bring your own service or leverage native cloud services

3 Share services at different locations*

4 Ability to continue to use cloud native services (Elastic Load

Balancer)

5 API driven, scalable, agent-less

6 Seamless overlay of policy

* Full service automation and shared services are part of roadmap

 September 2019

ACI Extensions to Azure ACI 4.2

Site 1 Site 2
On-Prem DC Public Cloud
MSO

Azure
Region

VXLAN / BGP EVPN Infra VNET

IPSec VPN Tunnel (Underlay) CSR 1000v

VM VM VM Express Route
User VNET2 User VNET2
IP Network

Common Discovery Policy Monitoring and Single point Operational

governance and visibility translation troubleshooting of orchestration consistency
 September 2019

ACI Extensions to Azure (Cont.) CI 4.2

Site 1 Site 2
On-Prem DC MSO Public Cloud

Azure
Region

EPG EPG ASG SG ASG ASG ASG

Contract Contract EPG DB
Web App Web Rule App Rule DB

Common Discovery Policy Monitoring and Single point Operational

governance and visibility translation troubleshooting of orchestration consistency
 September 2019

ACI Multi Cloud ACI 4.2

MSO

Site 1 Site 2 Site 3

VM VM VM VM VM VM VM VM VM

Region: us-east-1 Region: UK South Region: ap-northeast-1

Multi-Cloud with AWS and Azure Cloud Sites supported w/o ACI Fabric on-Prem
With Out-of-Band MSO
 February 2019

Micro Segmentation with ACI ACI 4.1

Sources of policy Segmentation

App group Dev zone

OpenShift Sales apps
Kubernetes
DB group Test zone

CI/CD
workflow HR apps
Web group Prod zone
Cloud
Center
VMM
Cisco APIC Application
components
Application
groups
Application
zones

Author Enforcement
Write me a email:
ratko.perlic@flintmail.com
Final evaluation:
https://tinyl.io/4UGX