1

2
3
It’s always been true, but not always discernable, but NW architecture and the DC
environment is a team sport; no one individual can do it all. The Application
Administrator might take care of load-balancing, Security takes care of Firewalls,
Network team takes care of routing/switching.
These administrators all use the same network, but each has a different view,
different requirements – and language. ACI provides a common policy and
operations framework, while at the same time provides these teams their different
views and needs.

4
In ACI you have a stateless pool of resources and all the players work from the same
tool.

5
Application Policy Infrastructure Controller (APIC) is the policy controller. The APIC is
not in the data path nor in the control path. It contains the management configuration
and is the policy repository.
You order in sets of three, provides a highly redundant cluster of three Servers at
FCS.
Factors associated with Scalability: Number of Spines, Leaf, Policies, Endpoints,
Contracts, Telemetry Data retention, etc..

6
7
The APIC is a purpose-built C-Series rack-mount server, including SSDs and VICs.
The current support is RMA (return merchandise authorization) the whole server,
even if an individual component fails – an adapter, drive, etc..
APIC consists of a Management API; any sort of Restful client can interface with
APIC.
The APIC has four different functionalities:
•  Policy
•  Topology management
•  Observer – stats and health monitoring
•  Boot – Node IP address management is provided by the APIC internal DHCP
service. All nodes will be assigned an IP address and each will be classified as a
VxLAN Tunnel End-Pt (VTEP – discussed in detail in a later lesson). Software
loads is managed by APIC for itself and all switch nodes (leaves and spines).

Scale-out processing (N+2 replication – this distributed datastore refers to the fact we
have 3 APICs; roadmap may have up to 31-APICs. Today the minimum and
maximum is 3. The datastore is replicated across all APICs; you have an original + 2
copies.

8
The Observer collects statistics, events, faults, and logs.
Pre-determined weighting system – observer takes in all these different measures
and observations, then renders a number for the Health Score.
Along with the global health score, each Tenant has its own health score.
Administrators can determine health scores for a given context. The idea is to isolate
based on function (i.e. state, drops, etc..)
Note: Customers may see a low health score from their fabric if ports with SFPs
plugged-in haven’t been configured yet.

9
Multi-tenancy implies partitioning of some kind – traffic, services, and mgmt. ACI’s
first level of partitioning is the ‘Tenant’. Based on this being a logical model, and this
hierarchy, you can specify at different levels where you want visibility (white vs black
vs fabric etc..) and lock it at different layers based on functionality or role you want to
provide.

In the course Labs, the first task is configure a Tenant; then create all application
components under that Tenant.

10
The controller (APIC) uses in-band connectivity for node communications. ACI
exploits the capabilities of LLDP which allow APICs and Leaves to discover each
other. All nodes require an administrator to validate the serial numbers and provide a
names. You can validate the serial numbers upfront, prior to physical connectivity.
The ‘Infrastructure VRF’ is used as the inband management; the IP is not routable
outside the fabric. This allows separation of ‘Tenant’ traffic within the fabric.

Policy Element Inter-Fabric Messaging/Comms to be established. The

encrypted TCP session is initiated from the APIC to the node when the node is
listening on TCP port 12183

Note: the Lab uses a simulator to demo initial setup.

11
APIC has four functions – Policy, Topology, Observer, Boot. These functions are
spread (shard) across all three APICs. The ‘shards’ are only sized based on three
APICs; that’s the replication factor – N+2 replication. We have a Preference w/ two
copies.
The slide shows a 3-node cluster. Each shard will touch 3-APICs.
The ‘Preference’ designation specifies the preferred copy (shard); ‘1’ is highest.
Writes are made to the highest preference then replicated-out. If Shard-1 goes away,
#2 is the highest available shard and write are made to 2 then replicated to 3. When
the former shard-1 comes-back online, it’ll sync the database.
The shard preference dictates where ‘Writes’ happen -- to the highest preference
reachable.

Sharding provides scaling (Spread out processing)

Replication provides fault tolerance (each shard has two copies)

Each shard has 3 replicas

•  Shards evenly distributed
•  No data is lost as long as only 1 or 2 APIC appliances die.

The message of behind this slide – modern resilient databases (MTBF of N+2
is better than N+1)

12
RESTful over HTTPs:
•  JSON + XML
•  Unified: automatically delegates request to corresponding components
•  Transactional
•  Single Management Entity yet fully independent components

Object Oriented
•  Comprehensive access to underlying information model
•  Consistent object naming directly mapped to URL
•  Supports object, sub-tree and class-level queries

In ACI, everything (GUI, scripts) is done over the API. The NOXS construct on the
switches was written on top of the API. This is similar to UCS – all interfaces (GUI,
CLI) go through the API.

REST (representational state transfer) – it’s an architectural style consisting of a

coordinated set of architectural constraints applied to components, connectors, and
data elements, within a distributed hypermedia system. For example, looking at how
Web application behaves: a network of web pages (a virtual state-machine), where
the user progresses through an application by selecting links (state transitions),
resulting in the next page being transferred to the user and rendered for their use
(representing the next state of the application)."
REST ignores the details of component implementation and protocol syntax in order

13
Everything is manipulated by REST. Every object in the APIC can be addressed by a
URL; which is a Characteristic of a REST based network. In REST, ‘Named
resources’ – are a system is comprised of resources which are named using a URL.
•  REST can do Post, Gets, Deletes and Update.
•  You can see in the HTTPS URL whether you’re using json or xml.

/api/ —Specifies that the message is directed to the API.

mo | class —Specifies whether the target of the operation is a managed object (MO)
or an object class.
Dn —Specifies the distinguished name (DN) of the targeted MO.
className —Specifies the name of the targeted class. This name is a concatenation
of the package name of the object queried and the name of the class queried in the
context of the corresponding package. For example, the class aaa:User results in a
className of aaaUser in the URI.
json | xml —Specifies whether the encoding format of the command or response
HTML body is JSON or XML.

14
15
16
APIC Hardware Ports –

1 Console port for a direct connection to a console

2 Cisco Integrated Management Controller (CIMC) port (alternative) port dedicated to
a console (1-Gigabit Ethernet connection) connection
3 Out-of-band management ports (can also be used as CIMC ports)
4 Virtual Interface Card (VIC) for either optical (VIC1225) connections or 10GBASE-T
(VIC1225T) connections
5 Two fiber optic or 10GBASE-T ports for connections to a downlink port on one or
two TOR switches

17
18
•  Fabric interfaces are configured as Linux bond interfaces with active/standby
failover
•  OOB management interfaces are configured as Linux bond interfaces with active/
standby failover
•  OOB management interfaces’ default setting is configured as shared with CIMC
•  A dedicated CIMC interface can be activated if CIMC is configured in dedicated
mode

19
20
On first boot, APIC console presents initial setup options
•  Fabric name
•  Cluster configuration (# APICs, unique ID per APIC device [1,2 or 3])
•  IP address pools for fabric (Infra VRF tunnel endpoints, multicast)
•  Make sure this does not overlap with any subnet you expect to
reach via APIC OOB.
•  Out-of-band management configuration
•  Admin user configuration
•  Fabric node membership
•  Pre-assign names based on serial numbers (text file import)
•  or assign names manually after fabric discovery

Note:
GIPO -- Group IP Outside
AVS – Application Virtual Switch (ACI for 1000v)

21
APIC and Fabric Node Connections -

•  Connect leafs to spines to build fabric

•  Connect each APIC to two leafs (PortChannel auto-config’d)
•  Connect OOB Management Network – APIC and switch node dedicated
management ports
•  Ready for APIC config and fabric bring-up

22
First Switch Bring-up
(a) Switch comes up in booting state
– Don’t relay/switch packets
– Send LLDP, DHCP resp packets to Sup
– All ports in L3 mode
– LLDP enabled on all ports
– Only required features enabled
(b) Detects ports connected to other switches/APIC (via LLDP)
– Link local IP assignment
– Start DHCP on LLDP validated ports
(c) Sends DHCP request
(d) APIC responds with TFTP server address and install script location
– Leaf installs host-route for TFTP server pointing to neighbor
(e) Switch downloads and executes the install script. Install script
– Downloads and installs switch image
– Downloads Infrastructure Policy (Controller IP, Overlay VLAN, Wiring Plan,
etc)
(f) Switch reboots
- Configures as DHCP relay for additional switch discovery

Policy Element Inter-Fabric Messaging/Comms to be established. The

encrypted TCP session is initiated from the APIC to the node when the node is
listening on TCP port 12183

23
24
Firmware management – ACI uses policies for image management. The image
repository in APIC will check versions on the Leafs, spines etc.. and upgrade
accordingly.
You can use a policy to change the management IP address.

25
The different Management networks in ACI:
•  Infrastructure VRF – is used as the inband mgmt. The IP is not routable outside
the fabric. This is how we achieve isolation of ‘Tenant’ traffic within the fabric.
•  Inband management – (none by default) assign IP address much like you would
use loop-back interface on a router, or a management VSAN on MDS.
•  OOB – assigned to dedicated management ports on the switches.

26
27
Basic Elements – Main Navigation
Two Level Top Navigation
•  Main Sections
o  System
o  Tenants
o  Fabric
o  VM Networking
o  L4-L7 Services
o  Admin

28
29
30
31
32
•  Hierarchical Organization
•  Folders/Tree Nodes
•  Context Menu
•  Workspace syncs with navigation tree
•  Consistency right-click on tree and Action button

33
•  Pagination Controls
•  Sort + Filter (same control location as Windows Tables)
•  Auto update (websocket)
•  Download to XML

34
35
36
37
38
39
40
41
42