Nutanix Enterprise Cloud Administration (ECA)

Nutanix Certified Professional - Multicloud Infrastructure (NCP-MCI)

Muhammad Magdy
www.linkedin.com/in/muhammad-magdy-996323131

1
 Module (01)
Introduction to Nutanix
❖ Hyperconverged Infrastructure (HCI):
- Hyperconverged Infrastructure (HCI) is essentially a set of servers (for compute, storage) and a
hypervisor. Within HCI, each server (called a "node") participates as a part of a cluster of nodes
providing both storage capacity and resiliency.
- HCI is both hardware and software. The underlying hardware is interchangeable, as long as its
general capabilities can support whatever workloads you intend to operate.
- This hardware is powered and managed by a distributed software layer that eliminates the
common pain points associated with three-tier infrastructure or legacy infrastructure (separate
storage, storage networks, and servers).
- The software running on each node distributes all operating functions across the cluster for
superior performance and resilience.
- All nodes include flash to optimize storage performance, and all-flash nodes are available to
deliver maximum I/O throughput with minimum latency for all enterprise applications.
- With HCI, when the time comes to retire nodes and replace them with newer ones, you simply
add the new node to your cluster, let it integrate, and then tell the cluster’s admin console that an
old node is being taken out of service.
- The benefits of HCI can be broadly classified into three types:

❖ Nutanix Overview:
- Nutanix provides a single platform that unifies hybrid multicloud management. The Nutanix cloud
platform integrates compute, virtualization, storage, networking, security, and containers, and
simplifies day to day management of a company’s IT environment.

2
- The Nutanix cloud platform also extends from private to public clouds, currently available on
Amazon Web Services, with support for Microsoft Azure under development.
- The Nutanix Cloud Platform is built on our industry-leading HCI and:
• Delivers a consistent operating model across public, private, and hybrid clouds.
• Supports any application and workload in any location.
• Offers choice and flexibility so that businesses can implement the right cloud operating
model for them.
- Nutanix solutions are available in five solution packages that bring together the infrastructure
management, operations, security, and business continuity technologies:

• Nutanix Cloud Infrastructure (NCI): Provides software solution including compute, storage
and networking for VMs and containers, that can be deployed on-prem or in public clouds.
Solution package includes the following products:
▪ AOS Storage: A highly automated, hyperconverged storage solution that scales with
your data.
▪ AHV Virtualization: A secure virtualization solution that streamlines operations, and
does not have to be licensed, deployed, and managed separately.
▪ Nutanix Kubernetes Engine: It allows you to fast-track your way to production-ready
Kubernetes and simplify lifecycle management.
▪ Nutanix Disaster Recovery: To deploy and manage DR solution that allows you to be
proactive with your DR plans and minimize downtime and data loss, whether you are
replicating to an on-prem DR site or the cloud.
▪ Flow Network Security: It allows to automate and protect against cyber threats by
creating software-based firewalls for critical apps and data without the management
overhead.

3
 ▪ Nutanix Cloud Clusters (NC2): It reduces the operational complexity of migrating,
extending or bursting your applications and data between on-premises and clouds.
• Nutanix Cloud Manager (NCM): Delivers operations including monitoring, insights, and
automated remediation to deploy, operate, and manage their applications. Drives financial
accountability & cost governance with intelligent resource optimization. Unifies security
operations for workloads and data across clouds, automating incident response with
intelligent analysis and regulatory compliance. Solution package includes the following
products:
▪ Intelligent Operations: (Formerly Prism Pro/Ultimate), It optimizes capacity, proactively
detects performance anomalies, and automates operations tasks with ease and
confidence.
▪ Self Service: (Formerly Nutanix Calm), It streamlines how teams manage, deploy and
scale applications across hybrid clouds with self-service, automation and centralized role-
based governance.
▪ Cost Governance: (Formerly Nutanix Beam), It allows you to drive financial
accountability with intelligent resource sizing and accurate visibility into cloud metering
and chargeback.
▪ Nutanix Security Central: It allows you to unify security operations for your workloads
and data on any cloud type while automating incident response with intelligent analysis
and regulatory compliance.
• Nutanix Unified Storage (NUS): Delivers software defined storage for multiple protocols
(volumes, files, objects) to support a variety of workloads deployed anywhere. A single point
of management for all storage resources eliminates complexity of multiple interfaces.
Solution package includes the following products:
▪ Files Storage: Allows you to centrally manage, scale and adapt to changing file-storage
needs from on-premises to multiple clouds.
▪ Volumes Block Storage: Bridges physical and virtual infrastructure, combining them
into one unified platform.
▪ Objects Storage: Delivers secure S3-compatible object storage at massive scale to
hybrid cloud environments.
▪ Mine Integrated Backup: Minimize downtime and consolidate your backup operations
into one turnkey solution.

• Nutanix Database Service (NDB): Simplifies database management across hybrid

multicloud environments for database engines like Microsoft SQL and Oracle Database with
powerful automation for provisioning, scaling, patching, protection, and cloning of database
instances. Solution package includes the following product:
▪ Nutanix Database Service: Easily operate fleets of Microsoft, MongoDB, MySQL, Oracle,
and PostgreSQL databases at scale on-premises and in the cloud.

4
 • Nutanix End User Computing Solutions (EUC): Delivers virtual apps and desktops to users
worldwide from public, private, and hybrid cloud infrastructure. Solution package includes
the following products:
▪ Virtual Desktop Infrastructure (VDI): A complete software stack to unify your hybrid
cloud infrastructure including compute, storage, network, and hypervisors, in public or
private clouds.
▪ Frame (Desktop-as-a-Service): Deliver virtual apps and desktops to users, either in the
public cloud with AWS, Azure, or GCP, or on-prem with Nutanix AHV.

❖ Nutanix Core Components:

- Three products form the core of the Nutanix solution, so much so that they are part of every
license tier for the Nutanix Cloud Infrastructure (NCI) solution package. These three products are:

• AOS Storage: It is a scalable, resilient, high performance, distributed storage solution that
you can use for all of your workloads while eliminating the need for separate SAN and NAS
solutions. It also includes a comprehensive set of capabilities for performance acceleration,
data reduction, data protection, and more.
• AHV Virtualization: It is a comprehensive virtualization solution that has been hardened to
meet the strongest security requirements, and offers a variety of features including
Intelligent VM placement, live migration, hypervisor conversion, and cross-hypervisor high
availability.
• Prism: It lets you manage your environment from a single console & simplify monitoring
and remediation with an end-to-end, application-centric view of your network from every
node in the cluster to VM-specific details. Maintain control over resources with role-based
access control (RBAC), and automate workflows with Prism’s comprehensive REST APIs.

5
 ❖ Understanding Nutanix Cluster:
- Node:
• A node is an x86 server with compute and storage resources. A single Nutanix cluster
running AHV can have a maximum of 32 nodes.
• A single (non-mixed) cluster running ESXi can have a maximum of 48 nodes, while a single
Hyper-V cluster can have a maximum of 16 nodes.
• There are 3 types of nodes are available:
▪ HCI Node: the most common type of node. It includes CPU, memory, and storage in a
single physical chassis.
▪ Storage-only (SO) Node: It has the bare minimum CPU capacity, but a significant
amount of onboard storage.
▪ Compute-only (CO) Node: It has the bare minimum onboard storage, but a significant
amount of CPU and memory.
iNote:
* A HCI node can run any supported hypervisor: Nutanix AHV, VMware ESXi, or Hyper- V but SO
and CO nodes support AHV only.
- Block: It is a chassis that holds 1 to 4 nodes, and contains power, cooling, and the backplane for
the nodes. The number of nodes and drives depends on the hardware chosen for the solution.
- Cluster:
• It is a logical grouping of physical and logical components that can consist of one, two,
three, four, or more nodes, and these nodes can be housed in one or more blocks.
• Since a cluster is both a physical and a logical grouping, it is possible for nodes in a single
block to belong to different clusters.
• Joining multiple nodes in a cluster allows for resources to be pooled. As an example, all
storage hardware in a cluster is presented as a single storage pool.

❖ Nutanix Cluster Components:

- The Nutanix cluster has a distributed architecture, which means that each node in the cluster
shares in the management of cluster resources and responsibilities.
- Within each node, there are software components (also known as AOS Services) that perform
specific tasks that are essential to cluster operations.
- All components run on multiple nodes in the cluster and depend on connectivity between their
peers. Most components also depend on other components for information.
- The following figure is a detailed list of all cluster components. It highlights the role that each
component plays and the connections and relationships between components:

6
• Acropolis:
▪ The Acropolis follower runs on every CVM with an elected Acropolis leader and
responsible for stat collection and publishing, and provides VNC proxy capabilities.
▪ The Acropolis leader is responsible for stat collection and publishing, task scheduling and
execution, VM placement and scheduling, network controller, and VMC proxy.
• Genesis:
▪ It is a process which runs on each node and is responsible for any services interactions
(start/stop/etc.) and the initial configuration.
▪ It runs independently of the cluster and does not require the cluster to be configured or
running.
▪ The only requirement for Genesis to be running is that Zookeeper is up and running.
• Zookeeper:
▪ It stores information about all cluster components (both hardware and software), including
their IP addresses, capacities, and data replication rules, in the cluster configuration.
▪ It is active on either 3 or 5 nodes, depending on the redundancy factor (number of data
block copies) applied to the cluster.
▪ It uses multiple nodes to prevent stale data from being returned to other components. An
odd number provides a method for breaking ties if two nodes have different information.

7
 ▪ Of these nodes, Zookeeper elects one node as the leader. The leader receives all requests
for information and confers with its follower nodes. If the leader stops responding, a new
leader is elected automatically.
▪ Zookeeper has no dependencies, meaning that it can start even if no other cluster
components are running.
• Zeus:
▪ It is an interface to access the information stored within Zookeeper and is the Nutanix
library that all other components use to access the cluster configuration.
• Medusa:
▪ Distributed systems that store data for other systems (for example, a hypervisor that
hosts VM) must have a way to keep track of where that data is. In the case of a Nutanix
cluster, it is also important to track where the replicas of that data are stored.
▪ Medusa is abstraction layer that sits in front of the database that holds metadata. The
database is distributed in a ring topology across multiple nodes in the cluster for
resiliency.
• Cassandra:
▪ It stores all metadata about the guest VM data in a Nutanix storage container using a
modified form of Apache Cassandra.
▪ It runs on all nodes of the cluster. These nodes communicate with each other once a
second using the Gossip protocol, ensuring that the state of the database is current on
all nodes.
▪ Cassandra depends on Zeus to gather information about the cluster configuration.
• Stargate:
▪ A distributed system that presents storage to other systems (such as a hypervisor) needs
a unified component for receiving and processing data that it receives. The Nutanix
cluster has a software component called Stargate that manages this responsibility.
▪ All read and write requests are sent across an internal vSwitch to the Stargate process
running on that node.
▪ Stargate depends on Medusa to gather metadata & Zeus to gather cluster configuration
data.
▪ From the perspective of the hypervisor, Stargate is the main point of contact for the
Nutanix cluster.
• Curator:
▪ A Curator leader node periodically scans the metadata database and identifies cleanup
and optimization tasks that Stargate should perform.
▪ Curator shares analyzed metadata across other Curator nodes.
▪ It depends on Zeus to learn which nodes are available, and Medusa to gather metadata.
Based on that analysis, it sends commands to Stargate.

8
 ❖ Understanding Nutanix Prism:
- Nutanix Prism provides central access for administrators to configure, monitor, and manage
virtual environments by combining several aspects of datacenter management into a single, easy-
to-use solution.
- Prism is a part of every Nutanix deployment and has two core components:

• Prism Element:
▪ It’s a service built into the platform for every Nutanix cluster deployed.
▪ Provides the ability to fully configure, manage, and monitor Nutanix clusters running any
hypervisor like AHV, ESXi, Hyper-V, and so on.
▪ With multiple Nutanix clusters, each cluster has a unique instance of Prism Element for
the management of that specific cluster.
▪ Prism Element provides you with several different management capabilities for the
administration of a single cluster:
o Cluster Management > It involves configuring and monitoring the entities within
the cluster, such as virtual machines, storage containers, hardware components,
and so on.
o Storage Management > It monitors storage usage across the cluster and provides
the options to create storage containers & volume groups, configure a threshold

9
 warning for storage capacity available and reserve storage capacity for rebuilding
failed nodes, blocks or racks.
o Network Management > It tracks and records networking statistics for a cluster.
iNote:
* You can log into Prism Element using the cluster virtual IP address or the IP address of any
Nutanix Controller VM (CVM) in the cluster.

• Prism Central:
▪ It manages different clusters on one screen. These clusters can be in a single physical
location or in multiple physical locations.
▪ Offers the ability to enable a variety of services, such as Calm, Karbon, Files, and so on.
▪ Unlike Prism Element, which is built into every cluster, Prism Central is an application that
must first be deployed as a VM before it can be used.
▪ Even if you only have a single cluster, management via Prism Central gives you access to
features that are not present in Prism Element, such as advanced reporting capabilities;
the ability to create custom dashboards; tools to analyze system activity, plan for
resource needs, and automate routine administrative tasks; and more.
▪ Prism Central includes a host of other advanced features (based on your licensing) which
help in managing and monitoring your clusters.
o Cost Management > It provides access to a cost governance and security
compliance SaaS product offering, called Xi Beam. This helps gain visibility into
cloud spend and security compliance status across multiple cloud environments.
o Resource Planning > It analyze current and potential resource needs. The Capacity
Runway tab allows you to view current resource runway information across the
registered Nutanix and Non-Nutanix clusters.
o Task Automation > The X-Play feature allows automate routine administrative
tasks, and auto-remediate issues by creating Playbooks.
iNote:
* You can access Prism Central from the Prism Element home dashboard if the cluster is registered
to a Prism Central instance.
* Certain cluster specific details - such as the cluster virtual IP and the ISCSI data services IP -
cannot be configured in Prism Central. They can only be configured in the Prism Element.

❖ Nutanix Pulse Overview:

- Pulse is a feature that provides diagnostic system data to Nutanix support, allowing proactive
support to be delivered for Nutanix solutions by collecting the data in the background with little to
no effect on system performance.
- When Pulse is enabled, it sends a summary email of the cluster configuration to a Nutanix
Support server daily by default. These messages are sent through ports 80/8443/443 (by default) or
through your mail server (if configured).

10
- It is recommended to enable Pulse unless providing cluster information to Nutanix customer
support violates your security policies.
- Pulse only shares basic, system-level information. This includes:
• System alerts.
• Current Nutanix software version.
• Nutanix processes & CVM information.
• Hypervisor information, such as type and version.
- Important data, such as system-level statistics and configuration information, is collected more
frequently so that issues can be detected automatically, and troubleshooting can be made easier.

❖ Initial Cluster Setup:

- When you get access to a brand-new cluster, there are a number of basic setup tasks that you will
typically need to perform:
• Running Nutanix Cluster Check (NCC):
▪ Nutanix Cluster Check (NCC) is cluster-resident software that help diagnose cluster
health and identify configurations qualified and recommended by running hundreds of
checks and takes actions as necessary to resolve issues.
▪ Depending on the issue identified, NCC will raise an alert or automatically create a
Nutanix Support case.
▪ It is recommended to run NCC both before and after cluster upgrades, making changes
to cluster hardware, and after deploying a new cluster or completing the Foundation
process on an existing one.
• Configuring NTP Servers:
▪ Hosts and CVMs in a Nutanix cluster must be configured to synchronize their system
clocks with a list of stable NTP servers.
▪ Graphs in the Prism interface rely on CVM time, and incorrect time can skew these
graphs, especially in relation to other monitoring platforms such as vCenter.
iNote:
* Nutanix CVMs and the Prism Central VM default to UTC. And, in all versions from AOS 5.18 and
Prism Central pc.2020.8 onwards, timestamps in Nutanix logs are expressed in UTC irrespective of
the configured cluster time zone.

• Configuring SMTP Server:

▪ Nutanix systems (such as Prism Central and Prism Element) use SMTP to send Alert
emails, and to exchange emails with Nutanix Support.
• Configuring Authentication:
▪ Prism Central supports these user authentication options:
o AD or OpenLDAP authentication.
o SAML authentication.

11
iNote:
* If Identity and Access Management (IAM) is not enabled, the only supported IDP for single sign-on
is Active Directory Federations Services (ADFS). If IAM is enabled, more options are available,
including Azure ADFS, Okta, PingOne, Shibboleth, and Keycloak.
o Local user authentication.
iNote:
* Assigning an account the User Admin role will allow that user to view information in Prism
Central, perform any administrative task, and create and modify user accounts.
* Assigning an account the Prism Central Admin role will allow that user to view information in
Prism Central and perform any administrative task. However, a Prism Central Admin cannot create
and modify user accounts.
* If no role is assigned to a local user account, the user will be able to log into and view information
in Prism Central but will not be able to perform administrative tasks or manage user accounts

• Changing UI Settings:
▪ Both Prism Central and Prism Element allow you to modify UI settings, including login
screen animations, and timeout settings for admin and non-admin users.
• Configuring Welcome Banner:
▪ Prism Central allows you to configure a welcome banner, which is the first thing users will
see when they attempt to log in. The banner can include both a custom message and
custom graphics.

12
 Module (02)
Securing Nutanix Cluster
❖ Nutanix's Approach to Security:
- Nutanix integrates security into every step of product development, via an approach called the
security development lifecycle (SecDL) which is a fundamental part of product design.
- Nutanix conforms to RHEL 7 Security Technical Implementation Guides (STIGs). The Controller VM
supports STIG rules that are capable of securing the boot loader, packages, file system, booting
and service control, file ownership, authentication, kernel, and logging.
- With Nutanix Security Configuration Management Automation (SCMA), you can quickly and
continually assess and remediate your platform to ensure that it meets or exceeds all regulatory
requirements.
- SCMA checks multiple security entities for both Nutanix storage and AHV, automatically reports
and logs inconsistencies, and then reverts entities to the baseline as needed.
- Nutanix Security Advisories provide detailed information on the available security fixes and
updates, including the vulnerability description and affected product/version.

❖ Role Based Access Control (RBAC):

- Role Based Access Control (RBAC) in Prism Central allows you to use both built-in and custom
roles to define permissions at a granular level, and then assign those roles to users.
- By default, 10 built-in roles are available in Prism Central: Super Admin, Self-Service Admin, Prism
Admin, Project Admin, VPC Admin, Network Infra Admin, Prism Viewer, Operator, Developer, &
Consumer.
- If the built-in roles are not sufficient for your needs, you can create one or more custom roles.
- After you configure authentication, users and directories are not assigned permissions or roles by
default. Permissions must be explicitly assigned to users, authorized directories, or organizational
units.
- There are two ways in which you can assign roles to users or groups of users:
• Role Mapping:
▪ It allows you to use pre-defined roles in Prism Central - User Admin, Cluster Admin, and
Viewer - and assign those roles to users, groups, and organizational units.
▪ Role Mapping is typically used to create basic role maps.
• Role Assignment:
▪ If you are using custom roles in Prism Central, then you can assign those roles to users or
groups with the Role Assignment feature.

13
- Cluster RBAC allows you to give users or user groups access to one or more clusters that have
been registered with Prism Central. This feature will allow the user to take action on entities such as
VMs, containers, and hosts only on the clusters to which they have access.
iNote:
* Up to 15 clusters can be assigned to any one user or user group.
* Cluster RBAC is supported only on AHV and ESXi clusters.

❖ Cluster Lockdown:
- Nutanix supports the cluster lockdown feature which disable password-based authentication and
enforce keybased SSH access to the Controller VM and AHV on the Host. Only the default 'nutanix'
and 'admin' accounts can be used to access a CVM or host that are in lockdown.
- You can create one or more key pairs and add the public keys to enable key-based SSH access.
However, when site security requirements do not allow this, you can remove all public keys to
prevent SSH access.
iNote:
* Prism supports these key types: RSA & ECDSA.
* Deleting all the public keys and disabling remote login access locks down the cluster from SSH
access.

❖ Flow Network Security:

- Nutanix offers Flow Network Security which uses a policy-driven security framework that inspects
traffic that originates and terminates within the datacenter, eliminating the need for additional
firewalls.
- Flow also uses a workload-centric approach instead of a network-centric one. Security policies are
applied to categories. Through categories, policies inspect traffic to and from groups of VMs
without the policies having to be applied to the VMs themselves.
- After a VM is associated with a category, and that category is associated with a policy, traffic
continues to be monitored even if the VM migrates to a different network or changes its IP
address. Network attributes, such as IP address and VLAN, play no role in traffic monitoring.
- This is possible because Flow's security policies using application-centric policy language. This
means that you first need to specify the VMs that belong to the application you want to protect,
then identify the networks or entities with which you want to allow those VMs to communicate.
- You cannot specify categories and subnets that you want to block because the number of these
entities is typically much larger than the list of allowed entities, and tends to grow at a much faster
rate as well. Instead, once you specify what categories and subnets are allowed to communicate,
everything else is blocked.
- Policies can also be enforced in two ways:

14
 • Monitor Mode: It allows all traffic, including traffic that is not allowed by the policy. This
allows to get a better understanding of the policy's impact and refine it further as needed.
• Enforce Mode: Once a policy and its impact have been determined, then you can switch it
to Enforce mode, which effectively enables the policy, blocking all traffic that is not allowed
by its configuration.
- Flow offers 4 types of security policies:
• Application Security Policy: It allows you to select categories of VMs, specify allowed
traffic sources and destinations, and apply your policy only to those entities. All other traffic
will be blocked.
• Isolation Environment Policy: It allows you to block all traffic between two groups of VMs.
However, communication between VMs within a category will still be allowed.
• Quarantine Policy: It allows you to isolate a compromised or infected VM and optionally
want to subject it to forensics.
• VDI Policy: IT allows you to secure a VDI environment and to create security policies based
on users and groups in an AD domain. When users log into their virtual desktops, Flow's ID
firewall will place VMs in certain predefined categories automatically, ensuring that security
policies are automatically enforced.

❖ Data-at-Rest Encryption (DARE):

- Nutanix provides two options that you can use to secure your data:
• DARE using Self-Encrypting Drives (SEDs):
▪ This involves using a combination of SEDs and an external Key Management Server
(KMS) to secure data while it is at rest.
▪ When SED-based DARE is set up and configured correctly, the CVM cannot access data
on the drives if it cannot get the correct keys from the KMS.
▪ If a drive is stolen, data is inaccessible because the KEK cannot be obtained from the
drive itself. If a node is stolen, the KMS can revoke the node's certificates so that they
cannot be used to access data on the drives.
• DARE using Software-only Encryption:
▪ AOS uses the AES-256 encryption standard to encrypt your data. You can use the
Nutanix Native Key Manager (local & remote) or an external KMS to secure your keys.
▪ Data protection must be enabled for the cluster before data can be encrypted.
▪ A symmetric data encryption key (DEK) such as AES 256 is applied to all data being
written or read. The key is known only to AOS, so data cannot be accessed directly from
the drive.
- Some points to remember when enabling DARE are:
• On AHV, for software-only encryption, encryption can be implemented at the cluster level
only. On ESXi and Hyper-V, encryption can be implemented at the container level or the
cluster level.

15
 • Encryption cannot be disabled once it is enabled at a cluster level or container level.
• Encryption can be implemented on an existing cluster that contains data. Once the task to
encrypt a cluster begins, the operation cannot be cancelled.
• You can change the encryption from SED-based DARE to software-only encryption.
iNote:
* The SED-based DARE and software-based DARE options are available to Nutanix users with an
ultimate license.
- Encryption can be implemented through 6 steps:
• Step (1) SEDs are installed for all data drives in a cluster.
o These drives are FIPS 140-2 validated.
o An existing cluster can be converted by replacing existing drives with SEDs.
• Step (2) Data on the drives is encrypted, but read and write access is open.
o By default, the built-in manufacturer key is used to protect access to data. However,
when data protection for the cluster is enabled, the CVM must provide a proper key.
• Step (3) An DEK key, such as AES 256, is applied to all data being written or read.
o The key is known only to the drive controller and never leaves the physical
subsystem, so there is no way to access the data directly from the drive itself.
• Step (4) An KEK key is used to encrypt/decrypt the DEK and authenticate to the drive.
o Each drive has a separate KEK, generated by the FIPS compliant random number
generator in the drive controller.
o Although the KEK is generated on the node it is not stored locally; all KEKs are sent to
the KMS for secure storage and retrieval.
• Step (5) A leader encryption key (MEK) is used to encrypt KEKs.
• Step (6) Keys are stored in a KMS that is separate from the cluster.
o Each node maintains a set of certificates and keys to establish a secure connection
with the KMS.
o The CVM communicates with the key management server using the Key Management
Interoperability Protocol (KMIP) to upload and retrieve drive keys.
iNote:
* While only one KMS is required, multiple KMSes are recommended so that the KMS does not
become a single point of failure. KMSes should be configured in clustered mode, so that they can be
added to the Nutanix cluster as a single, resilient entity that is capable of withstanding a single
failure.

16
 Module (03)
Configuring Cluster Networking
❖ AHV Networking Terminology:
- Open vSwitch (OVS):
• It is an open-source software switch designed to work in a virtualization environment.
• AHV uses OVS to connect the CVM, the hypervisor, and user VMs to each other and to the
physical network on each node.
• Each AHV server maintains an OVS instance, and all OVS instances combine to form a single
logical switch.
• OVS behaves like a layer 2 learning switch that maintains a MAC address table.
• OVS supports many popular switch features, including VLAN tagging, Link Aggregation
Control Protocol (LACP), port mirroring, and quality of service (QoS).
- Bridge:
• It acts as a virtual switch to manage traffic between physical and virtual network interfaces.
• The default AHV configuration includes an OVS bridge called br0 and a native Linux bridge
called virbr0.
• The Linux bridge carries management and storage communication between the CVM and
AHV host. All other storage, host, and VM network traffic flows through the br0 OVS bridge.
- Virtual Switch (VS):
• A virtual switch is configured by default on each AHV node and virtual switch services start
automatically when you start a node.
• A VS defines a collection of AHV nodes and the uplink ports on each node.
• It’s an aggregation of the same OVS bridge on all the compute nodes in a cluster. For
example, vs0 is the default virtual switch, and is an aggregation of the br0 bridge of all the
nodes.
• The default virtual switch vs0 is created by the system and connects the default bridge br0
on all the hosts in the cluster during installation of or when upgrading to the compatible
versions of AOS and AHV.
• The default virtual switch, vs0, has the following characteristics:
▪ It cannot be deleted.
▪ The default bridges on all the nodes in the cluster map to vs0 then it is not empty. It
has at least one uplink configured.
▪ The default management connectivity to a node is mapped to default bridge br0 that
is mapped to vs0.
▪ The default parameter values of vs0 - Name, Description, MTU and Bond Type - can
be modified subject to aforesaid characteristics.
▪ The default virtual switch is configured with the Active-Backup uplink bond type.

17
- Ports:
• Ports are logical constructs created in a bridge that represent connectivity to the virtual
switch.
• Nutanix uses different port types, such as:
▪ Internal Port > It acts as the AHV host management interface. It is usually with the same
name as the default bridge (br0).
▪ Tap Port > It connects VM virtual NIC (vNIC) to the bridge.
▪ VXLAN Port > It’s only used for the IP address management (IPAM) functionality
provided by AHV.
▪ Bonded Port > It provides NIC teaming for the physical interfaces of the AHV host.
- Bonds:
• Bonded ports aggregate the physical interfaces on the AHV host for fault tolerance and load
balancing.
• By default, the system creates a bond named br0-up in bridge br0 containing all physical
interfaces.
iNote:
* Only use NICs of the same speed in the same bond.
• OVS bonds allow for several load-balancing modes to distribute traffic:
▪ Active-Backup:
o It’s the recommended and default bond mode, where one interface in the bond is
randomly selected to carry traffic and other interfaces in the bond are used only
when the active link fails.
▪ Balance-SLB:
o The balance-slb bond mode in OVS takes advantage of all links in a bond and
uses measured traffic load to rebalance VM traffic from highly used to less-used
interfaces.
o When the configurable bond-rebalance interval expires, OVS uses the measured
load for each interface and the load for each source MAC hash to spread traffic
evenly among links in the bond.
▪ Balance-TCP:
o Taking full advantage of bandwidth provided by multiple links to upstream
switches, from a single VM, requires dynamically negotiated link aggregation and
load balancing using balance-tcp.
o Nutanix recommends dynamic link aggregation with LACP instead of static link
aggregation due to improved failure detection and recovery.

18
- Bridge Chaining:
• In newer AOS versions, all AHV hosts use a bridge chain (multiple OVS bridges connected in
a line) as the backend for features like microsegmentation.
• Each bridge in the chain performs a specific set of functions. Physical interfaces connect to
bridge brX, and VMs connect to bridge brX.local.
• Traffic from VMs enters the bridge chain and flows through the chain. The bridge forwards
the traffic to the physical network or sends it back through the chain to reach another VM.
• All VM traffic must flow through the bridge chain, which applies microsegmentation and
network functions.
• The management of the bridge chain is automated, and no user configuration of the chain is
required or supported.
- IP Address Management (IPAM):
• IPAM enables AHV to assign IP addresses automatically to VMs using the DHCP.
• You can configure each virtual network and associated VLAN with a specific IP subnet,
associated domain settings, and group of IP address pools available for assignment.
• AOS uses VXLAN and OpenFlow rules in OVS to intercept DHCP requests from user VMs so
that the configured IP address pools and settings are used.
• A managed network refers to an AHV network in which IPAM has been enabled, whereas an
unmanaged network refers to an AHV network in which IPAM has not been enabled.
- Virtual Private Clouds (VPCs):
• A VPC is an independent and isolated IP address space that functions as a logically isolated
virtual network.
• A VPC could be made up of one or more subnets that are connected through a logical or
virtual router.
- Floating IP Address:
• A floating IP is an IP from the external subnet prefix that is assigned to a VM via the VPC
that manages the network of the VM.

19
 AHV Term ESXi Term Hyper-V Term
Virtual switch, logical
Bridge vSwitch, Distributed Virtual Switch
switch
Team or uplink port
Bond NIC team
profile
Port or tap Port N/A
VLAN tag or logical
Network Port group
network
Uplink pNIC or vmnic Physical NIC or pNIC
VM NIC vNIC VM NIC
Internal Port VMkernel port Virtual NIC
Active-backup Active-standby Active-standby
Route based on source MAC hash combined with Switch independent /
Balance-slb
route based on physical NIC load dynamic
LACP with Switch dependent (LACP)
LACP and route based on IP hash
balance-tcp / address hash

❖ Network Visualizer:
- On AHV clusters, a network visualizer is provided that displays a graphical representation of the
network in a cluster. You can use the visualizer to monitor the network and to obtain information
to identify network issues.
- It helps to view the physical and logical network topology, the number & types of devices in the
network, the network configuration of the devices in the topology and of components such as
physical and virtual NICs, as well as real-time usage graphs of physical and virtual interfaces.
- Before using the network visualizer, you need to:
• Configure network switch information on the Nutanix cluster.
• Enable LLDP or CDP on the first-hop switches.
o The visualizer uses LLDP or CDP to determine which switch port is connected to a
given host interface.
o If LLDP or CDP is unavailable, SNMP data is used on a best-effort basis.
• Optionally, configure SNMP v3 or SNMP v2c on the first-hop switches if you want the
network visualizer to display the switch and switch interface statistics.
o The visualizer uses SNMP for discovery and to obtain real-time usage statistics from
the switches.
iNote:
* Network visualization depends on a functional internal DNS system to map switch hostnames and
IP addresses based on the LLDP responses. Incorrect or partial DNS configuration displays
inaccurate network details in the UI.
* The network visualizer is only accessible through Prism Element only.

20
- Also, the network visualizer provides filter and grouping capabilities, allowing you to customize
the display and view only a specific set of networked devices and connections.
- The network visualizer can be divided into 2 panes:
• Virtual Networks Pane.
• Topology View Pane.

❖ Extending Subnets:
- A subnet can be extended between on-prem local and remote clusters or sites (Availability Zones)
to support seamless application migration between these clusters or sites.
- With Layer 2 subnet extension, you can migrate a set of applications to the remote AZ while
retaining their network bindings such as IP address, MAC address, and default gateway.
- The Layer 2 extension assumes that there are underlying existing layer 3 connectivity already
available between the Availability Zones.
- You can extend a Layer 2 subnet using:
• VPN: This can be used to extend the subnet across two Nutanix AZs.
• VTEP: This can be used to extend the subnet across two Nutanix AZs as well as between a
Nutanix AZ and one or more non-Nutanix datacenters.
- The supported configurations are:
• IPAM Type > Managed & unmanaged networks.
• Subnet Type > On-prem VLAN & VPC subnets.
• Traffic Type > IPv4 unicast traffic & ARP.
• Hypervisor > AHV & ESXi.
- Before configuring Layer 2 subnet extension between your on-prem AZs:
• Ensure that the Prism Central versions support Layer 2 virtual subnet extension.
• Ensure that you pair the Prism Central at the local AZ with the Prism Central at the remote
AZ to use the Create Subnet Extension wizard to extend a subnet across the AZs.
• Ensure that you set up a default static route with 0.0.0.0/0 prefix and the External Network
next hop for the VPC you use for any subnet extension.
• When using Nutanix IPAM ensure the address ranges in the paired subnets are unique to
avoid conflict between VM IP addresses across extended subnets.
• If connectivity between sites already provides encryption, consider using VTEP only subnet
extension to reduce encryption overhead.

iNote:
* Extend subnets between clusters managed by the same Prism Central.

21
 ❖ Segmenting Networks:
- Network segmentation enhances security, resilience, and cluster performance by isolating a
subset of traffic to its own network.
- The ways through which traffic can be isolated are:
• Isolating Backplane Traffic Using VLANs (Logical Segmentation):
▪ You can separate management traffic from storage replication (or backplane) traffic by
creating a separate network segment (LAN) for storage replication.
▪ This will create a separate network for backplane communications on the existing default
virtual switch and places the eth2 interfaces (that are created on the CVMs during
upgrade) and the host interfaces on the newly created network.
• Isolating Backplane Traffic Physically (Physical Segmentation):
▪ You can physically isolate the backplane traffic (intra cluster traffic) from the
management traffic (Prism, SSH, SNMP) into a separate vNIC on the CVM and using a
dedicated virtual network that has its own physical NICs.
• Isolating Service-specific Traffic:
▪ You can secure traffic associated with a service (for example, Nutanix Volumes) by
confining its traffic to a separate vNIC on the CVM and using a dedicated virtual network
that has its own physical NICs.
▪ This type of segmentation offers true physical separation for service- specific traffic.
iNote:
* Network segmentation for Volumes also requires you to migrate iSCSI client connections to the
new segmented network.
• Isolating Stargate-to-Stargate Traffic over RDMA:
▪ Some Nutanix platforms support remote direct memory access (RDMA) for Stargate-to-
Stargate service communication.
▪ You can create a separate virtual network for RDMA-enabled network interface cards. If a
node has RDMA-enabled NICs, Foundation passes the NICs through to the CVMs during
imaging.
- Unsegmented Network:
• In the default unsegmented network in a Nutanix cluster (AHV), the Controller VM has two
virtual network interfaces:
▪ Interface eth0: It is connected to the default external virtual switch, which is in turn
connected to the external network through a bond or NIC team that contains the host
physical uplinks.
▪ Interface eth1: It is connected to an internal network that enables the CVM to
communicate with the hypervisor.
• In the unsegmented network, all external CVM traffic, whether backplane or management
traffic, uses interface eth0. These interfaces are on the default VLAN on the default virtual
switch.

22
- Segmented Network:
• In a segmented network, management traffic uses CVM interface eth0 and additional
services can be isolated to different VLANs or virtual switches.
• In backplane segmentation, the backplane traffic uses interface eth2.

23
 Module (04)
Managing Images
❖ Image & Image Service:
- There are two types of images:
• ISO Image: It is a file that contains an image or a copy of the data on an optical disk.
• Disk Image: It is a virtual clone of a physical disk. The entire data, including the file
structure, files, and folders from the disk, are saved in the disk image.
- ISO and disk image files can be imported using Prism Central or Prism Element to install OS or
applications. With Prism Central you can clone an existing VM disk and add it to the image list.
- The image service is a feature available on AHV clusters that allows you to import and configure
operating system ISO and disk image files using either Prism Central or Prism Element.
iNote:
* The image service supports raw, vhd, vhdx, vmdk, vdi, ova, iso, and qcow2 disk formats.
* The image services used port 2007.
- Images that are uploaded to Prism Element, reside in and can be managed by only that Prism
Element. In essence, they are only available for use on a single cluster.
- If you are using Prism Central, you can migrate images manually to Prism Central and manage all
of the images for one or more clusters from a single location.
- An image migrated to Prism Central in this way remains on Prism Element, but can only be
updated, modified, and managed from Prism Central.

❖ Image Configuration in Prism Central:

- The Images dashboard is unique to Prism Central and has no equivalent in Prism Element. It
allows you to upload, import, and manage your images, as well as create and manage image
policies from a single page in Prism.
- There are 2 policies for images:
• Placement Policies: It enables you to configure policies that govern which clusters receive
the images that you upload by mapping images to target clusters using categories
associated with both those entities.
• Bandwidth Throttling Policies: It limits the bandwidth consumed during image creation
using the URL option in the specific clusters. Without this policy, image creation from a
remote server will consumes as much bandwidth as is available.
- When uploading an image, we have 3 options:
• Image File: This allows you to upload files directly from the workstation on which you are
accessing Prism Central.

24
 • URL:
▪ Most browsers impose file size limitations that affect the Image File method of upload.
▪ Also, the browser type, and CPU and RAM utilization on the workstation limit the number
of concurrent uploads.
▪ Concurrent uploads exceeding the default limit of the browser are queued or throttled
by the browser and can take more time.
▪ In these situations, or if you need to upload an image that is 2GB or larger, uploading
from a remote server is preferred.
• VM Disk: It allows you to select a VM that is managed by Prism Central and clone a VM disk
for use as an image.
- There are two ways you can determine where the image will be placed:
• Place image directly on clusters: This is recommended for smaller environments to select
one or more clusters; the uploaded image will be made available on all selected clusters.
• Place image using image placement policies: To use this option, you need to set up
image placement policies, which involves associating categories with clusters, and then
assigning categories to images. This is recommended for larger environments.
- Any image that you upload directly to Prism Element can be imported to Prism Central, so that
you can manage and update all your images from one, single location.

❖ Image Placement Policies:

- Prism Central allows you to define image placement policies that determine which clusters will
receive the images that you upload. These policies automatically assign images to target clusters
based on categories that are applied to images as well as clusters.
- You can also define how strictly these policies are enforced; 'Soft' enforcement allows clusters
that are not specified in the policy to use uploaded images whereas, 'Hard' enforcement prevents
this entirely.
- Creating an image placement policy involves three major activities:
• Create categories for images and clusters.
• Apply the created categories to images and clusters respectively.
▪ A category cannot be associated with entities from the Categories dashboard in Prism
Central.
▪ To assign categories, you need to navigate to the dashboard of the specific entity that
you want to associate.
• Configure the image placement policy.

25
 Module (05)
Creating Virtual Machines
❖ Creating & Managing VMs:
- You can use both Prism Central and Prism Element to create and manage VMs. However, since
Prism Element manages only a single cluster, it can be used to create and manage VMs on one
cluster only.
- With Prism Element, you can create, update, migrate, clone, power on or off, and delete VMs,
launch a VM's console and manage Nutanix Guest Tools. Some VM metrics can also be viewed on
Prism Element's VM dashboard.
- In addition to all of the functionality of Prism Element, Prism Central also allows you to manage
affinity and NGT policies, view filtered VM-specific lists of alerts and events, create VM templates,
enable/disable efficiency measurement and anomaly detection, run playbooks, manage categories,
add VMs to catalogs, manage VM ownership, and more. Prism Central also includes a larger
collection of VM metrics, available out-of-the-box.
- Creating a VM in either Prism Central or Prism Element is a straightforward process. However,
components such as networks, disks, categories, and so, which are required to build the VM need
to be setup or configured first.
- One of the prerequisites for creating VMs is VirtIO Drivers for Windows VMs:
• Nutanix VirtIO is a collection of drivers meant to enhance the stability and performance of
VMs. VirtIO bundles several different drivers, including ethernet adapter, RNG device, SCSI
pass- through controller, SCSI controller, serial driver, and so on.
• VirtIO is available in two formats:
▪ If you are performing a fresh installation of Windows on an AHV VM, use the VirtIO ISO.
▪ If you are updating VirtIO for Windows on an existing VM, use the VirtIO MSI file.
- AHV can run on hardware that supports UEFI and Secure Boot of UEFI, which allows you to enable
Secure Boot on user VMs running on AHV clusters.
• If you choose to enable UEFI, note that:
▪ Nutanix does not support converting a VM that uses IDE disks or legacy BIOS to VMs
that use Secure Boot.
▪ The minimum supported version of the Nutanix VirtIO package for Secure bootenabled
VMs is 1.1.6.

❖ Prism Self Service:

- Prism Self Service allows IT administrators to extend a measure of self-sufficiency and autonomy
to consumers of IT infrastructure.

26
- Individual users that need to create VMs with any degree of regularity can be empowered to do
so on their own, without the need for administrative intervention but this way does not give them
complete, unrestricted access to a cluster or its resources.
- Using a Self-Service feature known as Projects, an IT administrator can allow both users and self-
service administrators access to specific groups of resources, specific entities (such as VM
templates or images), and configure roles for project members.
iNote:
* Prism Self Service is supported on AHV clusters only.
- When configuring Prism Self Service, you can assign one of two roles to your users:
• Self-Service Administrator:
▪ A self-service administrator can create projects, add users or groups to projects,
configure roles for each project member which will grant those members access to
entities and actions on entities, publish VM templates and images to a catalog, monitor
project resource usage and adjust quotas as needed.
▪ Although self-service administrators do not have administrative access to Prism Central,
they do have full access to all VMs running on a cluster, including VMs that are not part
of a project. Also, after a user or group has been assigned the self-service administrator
role, the Prism Central administrator cannot limit the user or group's privileges.
• Project User:
▪ A project user is a user and consumer of resources.
▪ They are added to projects by either a Prism Central Administrator or a self-service
administrator, and can perform actions or work with entities based on the role and
permissions that have been assigned to them.
- Since self-service users can only create VMs based on pre-defined templates that are available in
Prism's catalog, an administrator (either the Prism Central Admin or the Self-Service Admin) must
make these VM templates available.
iNote:
* To enable your self-service users to deploy VMs on their own, it’s enough to “Add VM to Catalog”
option with no further changes or modifications.

❖ Nutanix Guest Tools:

- Nutanix Guest Tools is a set of software features installed in a VM and a Nutanix CVM. A Nutanix
Guest Agent publishes information about the VM to the Nutanix cluster, such as guest operating
system type, VM mobility status, and the Volume Shadow Copy Service (VSS).
- It comes bundled with AOS and can be installed in Windows and Linux VMs. The NGT software
package includes:
• The NGT installer, which allows you to install NGT in a guest VM.
• The Nutanix Guest Agent (NGA) service, which maintains communication between the CVM
and guest VMs.

27
 • The Nutanix VirtIO package, which includes VM mobility drivers that enable VM migration
between AHV and ESXi, in-place hypervisor conversion, and cross-hypervisor disaster
recovery.
- If you install NGT in a guest VM, you can use the following features:
• Self Service File Recovery: enables Nutanix administrators or VM owners to mount
snapshots directly to the VM they were taken from. This capability allows users to select
specific files from a snapshot.
• Nutanix VirtIO drivers: NGT installs Nutanix VirtIO drivers and provides the registry
configuration necessary to support the migration or conversion of VMs bidirectionally
between AHV and ESXi. AOS uses these drivers when performing Nutanix VM Mobility or
Nutanix cluster conversion. The required drivers are installed in Windows guests.
• Nutanix VSS hardware provider: enables integration with native Nutanix data protection.
The provider allows for application-consistent, VSS-based snapshots when using Nutanix
protection domains and is supported with Windows guests.
• Application-consistent snapshot for Linux VMs: It supports application-consistent
snapshots for Linux VMs by running specific scripts on VM quiesce.
iNote:
* For a detailed list of prerequisites and limitations related to NGT installation and management in:
- Prism Element, see the Nutanix Guest Tools Requirements and Limitations.
- Prism Central, see the NGT Management in Prism Central Requirements.
- Depending on whether you are using Prism Central or Prism Element, the process for enabling
and installing NGT is slightly different:
• In Prism Element, you need to first enable NGT on a VM, then log into the VM and install
NGT.
▪ It is important to note that you can only do this on one VM at a time.
• In Prism Central, allows you to both enable and install NGT directly from the management
console without having to launch the VM itself.
▪ If a VM is powered off or disabled, you will not be able to enable NGT, and will be
notified about this in Prism Central.
- NGT requires network connectivity between the Nutanix cluster and the guest VMs on port 2074
between the required VMs and the CVMs. NGT also requires an empty IDE CD-ROM slot in the
guest for attaching the ISO image.

28
 Module (06)
Managing Virtual Machines
❖ VM Migration:
- When live migration occurs automatically, it can be due to the Acropolis Dynamic Scheduler
(ADS) monitoring a cluster for hotspots and contentions and making adjustments as needed, or
due to a node being placed in maintenance mode.
- Automatic VM Migration with Acropolis Dynamic Scheduler:
• Acropolis Dynamic Scheduling (ADS) proactively monitors your cluster for any compute and
storage I/O contentions or hotspots over a period of time.
• If ADS detects a problem, it eliminates hotspots within a cluster by migrating VMs from one
node to another.
• ADS runs every 15 minutes and analyzes resource usage for that period of time.
• If the resource utilization of an AHV node remains > 85% for 15 minutes, migration tasks are
triggered to remove the hotspot.
• For a storage hotspot, ADS looks at the last 40 minutes of data and uses a smoothing
algorithm to use the most recent data.
• For a CPU hotspot, ADS looks at the average CPU usage over the last 10 minutes.
• ADS does not monitor memory and network usage.
- Automatic VM Migration During Maintenance Mode:
• When a node is put in maintenance mode, it is marked as unschedulable so that no new
VMs can be created on it.
• If VMs are already running on that node, an attempt is made to evacuate them.
• Upon exiting maintenance mode, all VMs are automatically returned to the original node, so
no manual intervention is needed.
• Some exceptions to this automated movement are agent VMs, VMs with GPUs, CPU
passthrough, PCI passthrough, and host affinity policies.
▪ GPU-enabled VMs can be live migrated, but manually by an administrator.
▪ Agent VMs are always shut down if a node is put into maintenance mode, and are
powered on once maintenance mode is exited.

❖ Affinity Policies:
- An affinity policy is an administrator-defined policy that determines which node or nodes of a
cluster a VM is allowed to or is restricted from running on.
- Two types of affinity policies can be defined in Prism:
• The VM-host affinity policy controls the placement of a VM. It is used to specify which
nodes a VM is allowed to run on. Every time you power on or migrate a VM to which this
policy is applied, the policy is checked and enforced.

29
 • The VM-VM anti-affinity policy is meant to keep specified VMs apart. That is, the VMs
specified in this type of policy will not be allowed to run on the same node. However, this is
a preferential policy and can be ignored by ADS if resource constraints require VMs to be
temporarily placed on the same node.
- Prism Central's affinity policies are category based, which means that you need to have either
defined and used custom categories or applied the default categories before your policy can take
effect.
- Even if a category does not have VMs or hosts associated with it, you can still create a VM-host
affinity policy, and assign categories later. Once you assign categories to the correct entities, you
can re-enforce the policy to move VMs onto their designated hosts.
- When you create an affinity policy, is it recommended to always include a category with multiple
hosts, or multiple categories that include multiple hosts. If you create an affinity policy with just
one host, the VMs that are part of that policy will not be moved to and powered on another host if
the original host fails.
iNote:
* VM-VM anti-affinity policies cannot be defined in Prism Central.
- Prism Element can be used to create both VM-host affinity policies and VM-VM anti-affinity
policies. However, they can only be applied to one VM at a time, either when the VM is being
created or when it is being updated.
- VM-VM anti-affinity policies can only be defined using the command line. To do this, you need to
SSH into the CVM, create a group, add VMs to it and then apply anti-affinity to the group.

❖ VM Storage Policies:
- By default, storage policies are applied to storage containers. Any entities that are present on
storage containers with these storage optimization features automatically inherit those features.
- However, if you want more control over the storage policies that are applied to your VMs, it is
possible to create storage policies and apply them via categories to a VM or groups of VMs.
- There are several considerations involved in creating storage policies:
• Storage policies can be created in Prism Central.
• Storage policies are supported only on AHV, not entities running on ESXi or Hyper-V.
• Categories are required for storage policies to be implemented. A storage policy can only be
associated with a category, and cannot be directly associated with a VM.
• Multiple categories can be associated with a single storage policy. However, a single
category cannot be associated with multiple storage policies.
• After encryption is enabled using a storage policy, it cannot be disabled. However, if the
storage policy is deleted, any new data that is written to entities will not be encrypted. Old
data, which was written while the policy was still in effect, will remain encrypted.

30
 ❖ Playbook Overview:
- The X-Play (pronounced cross-play) feature in Prism Central allows you to quickly and easily
automate common administrative tasks using playbooks that can be manually or automatically
triggered, and consist of a list of actions that the system will perform in order to complete a task or
resolve an issue.

❖ VM High Availability:
- VM High Availability (VM HA) ensures that VMs restarted on another AHV host in an AHV cluster
when a host becomes unavailable due to network partitioning, or a host management process
failure.
- VM HA also respects affinity and anti-affinity rules. Two VM HA modes are available, and can be
enabled either in Prism or by using the command line:
• Default Mode:
▪ This requires no configuration and is included by default when installing an AHV-based
Nutanix cluster.
▪ When an AHV host becomes unavailable, the VMs that were running on that AHV host
restart on the remaining hosts if the necessary resources are available.
▪ If the remaining hosts do not have sufficient resources, some of the failed VMs may not
restart.
• Guarantee Mode:
▪ This configuration reserves space throughout the AHV hosts in the cluster to guarantee
that all VMs can restart on other hosts in the cluster when an AHV host becomes
unavailable.

31
 Module (07)
Protecting Virtual Machines & Their Data
❖ Data Protection Strategies:
- To safeguard both VMs and their data, Nutanix provides data protection at the VM, volume
group, and file levels using protection domains and remote sites to implement a data protection
strategy.
- Several types of protection strategies are supported:
• Per-VM Backup: This involves specific VMs for backup to a different site and it is useful for
remote office and branch office (ROBO) environments.
• Selective, Bi-directional Replication: A flexible replication solution requires that replication
be bi-directional, from the primary site to the secondary site, and back to the primary site
when required.
• Synchronous Datastore Replication: Datastores can be spanned across 2 sites to provide
seamless protection in the event of a site disaster.
- Nutanix supports a variety of different topologies to meet a variety of requirements:
• One-to-many:
▪ In this scenario, there is one central site with multiple remote locations.
▪ The workloads on the central site can be replicated to the remote sites.
▪ In the event of a disaster, workloads can be restarted on either of the remote sites,
providing a high level of VM availability.

• Many-to-one:
▪ It is also called a hub and spoke architecture, in which workloads running on multiple
sites are replicated to a single, central site.
▪ Centralizing replication to a single site may improve operational efficiency for disperse
environments geographically.
• Many-to-many:
▪ In this topology, there is neither a single primary site nor a single backup site.
▪ Instead, multiple sites exist, and all serve as replication targets for each other.
• Two-way mirroring:
▪ In this topology, there are two sites, each of which serves as a replication target for the
other.
▪ There are active workloads running on both sites simultaneously, and there are no idle
resources in either location.

32
 ❖ Data Protection in Prism Element:
- When creating a protection domain, it is possible to perform both local and remote backups.
- Local Backup:
• It is available by default.
• The “Auto protect related entities” option means, for example, if we choose to protect
certain VMs, any volume groups associated with those VMs will automatically be added to
the protection domain.
- Remote Backup:
• To enable remote backups, a remote site must first be configured.
• A remote site can be either another on-prem cluster or a public cloud like AWS.
• You can enable proxy which allows the specified IP addresses (remote CVM IP addresses or
remote Virtual IP address) to be used as a proxy to communicate with a Nutanix cluster on
the remote site.
• In this case, the source cluster communicates with only one of the remote proxy IP
addresses (remote CVM IP addresses or remote Virtual IP address), which forwards the
requests to the appropriate remote CVMs.
• The proxy setting on the remote site limits the replication traffic to the defined destination
remote site IP address (many to one) rather than to each individual CVM IP address in the
destination cluster (many to many).
▪ The many-to-one replication approach can be used to simplify firewall rules
configuration.
• You can designate the site as:
▪ Backup allows the remote site cluster to be used as a backup replication target. This
means data can be backed up to this remote site cluster and snapshots can be retrieved
from the site to restore locally, but failover protection is not enabled.
▪ Disaster recovery allows the remote site to be used both as a backup target and as a
source for dynamic recovery, which means that failover VMs can be run directly from the
remote site.
• Bandwidth throttling allows you to define a policy that enforces a limit for the network
bandwidth, based on utilization of your network.

❖ Understanding NearSync:
- NearSync allows you to protect your data with an RPO of 1 to 15 minutes. NearSync is useful for
protecting critical applications, securing data with minimal loss, providing granular control during
the restore process, and allowing for the resolution of a disaster event within minutes.
iNote:
* Recovery Point Objective (RPO): A time interval that refers to the acceptable data loss if a failure
occurs. For example, if the RPO is 1 hour, the system creates a recovery point every 1 hour. In the
event of a recovery, you can recover VMs with data as of up to 1 hour ago.

33
- This is possible because NearSync uses a Nutanix technology called lightweight snapshots (LWS).
These snapshots are generated at the metadata level only and continuously replicate incoming
data generated by workloads running on the active cluster.
- When you configure NearSync on a protection domain, snapshots will first be generated on an
hourly basis and replicated in order to seed the remote site with data.
- Once the required snapshots have been generated and replicated for the purpose of data
seeding, LWS snapshots will be generated and replicated instead based on the RPO defined in the
protection domain.

❖ Understanding Asynchronous Replication:

- Asynchronous replication refers to schedules that have been configured with an RPO of 60
minutes or more. These schedules involve full snapshots, unlike the lightweight snapshots that are
used for NearSync.
- There is no protection method that allows for an interval of 15 to 59 minutes. For RPO of 15
minutes or less, NearSync is used. For RPO of 60 minutes or more, async replication is used.

❖ Understanding Cloud Connect:

- Cloud Connect is a Nutanix data protection capability that facilitates asynchronous replication to
Amazon AWS.
- When AWS is configured as a remote site for backup, a single node cluster with a Nutanix CVM is
created on an AWS cloud in the region of your choice.
- This single node cluster has a 30 TB disk attached to it, with a usable capacity of 20 TB. Once
configured, the cluster on the remote site can be managed in Prism Element just like any other
remote site.
- Amazon S3 is used to store data (extents) and Amazon Elastic Block Store (EBS) is used to store
metadata. You can then use the Amazon management tools to manage billing and related usage.
You will be charged only for used capacity, not the full capacity of the single node cluster.
- When the AWS Remote feature replicates a snapshot data to AWS, the Nutanix Controller VM on
AWS creates a bucket on S3 storage.

❖ Understanding Metro Availability:

- Metro availability is a policy applied on a storage container, which spans the storage container
across two sites by pairing a storage container on the local cluster with one on a remote site and
then synchronously replicating data between the local (active) and remote (standby) storage
containers.
iNote:
* Metro availability configurations can include VMs, but they cannot include volume groups.

34
- Metro availability policies apply per storage container (not cluster), so a cluster can be active for
one datastore and standby for another.
- In addition, metro availability storage containers can co-exist with regular storage containers in
the same cluster.
- Metro availability is supported on clusters running ESXi or Hyper-V only. Metro is supported on
ESXi clusters that include storage-only nodes (which run AHV) but not Hyper-V clusters with
storage-only nodes.
- When configuring metro availability, you can also configure a Witness VM.
• It is a special VM that monitors the metro availability deployment's configuration health.
• The Witness VM resides in a separate failure domain to provide an outside view that can
distinguish a site failure from a network interruption between the metro availability sites.
• The main functions of a Witness include:
▪ Making a failover decision in the event of a site or inter-site network failure.
▪ Avoiding a split-brain condition where the same storage container is active on both sites
due to (for example) a WAN failure.
▪ Handling situations where a single storage or network domain fails.

❖ Leap Overview:
- Protection domains are a VM and VM-data protection mechanism available in Prism Element.
However, Prism Central offers Leap.
- Leap protects your guest VMs and orchestrates DR to other Nutanix clusters (or Xi Cloud Services)
when events causing service disruption occur at the primary site.
- Protection policies with Asynchronous, NearSync, or Synchronous replication schedules generate
and replicate recovery points to other on-prem or Xi Cloud sites.
- Recovery plans orchestrate DR from the replicated recovery points to other Nutanix clusters at
the same or different on-prem sites.
- There are two variants of this Nutanix data protection capability:
• Leap is used to facilitate VM, data, and application protection and DR to other Nutanix
clusters at the same physical site or different physical sites.
• Xi Leap is an extension of Leap in which VM, data, and application protection extends from
an on-prem site to Xi Cloud Services or from Xi Cloud Services to an on-prem site.
- How Does Leap Work?
• Leap uses categories to group guest VMs and automate the protection of guest VMs as
applications scale.
• Application recovery is flexible with network mappings, an enforceable VM start sequence,
and inter-stage delays.
• Application recovery can also be validated and tested without affecting your production
workloads.

35
 • Asynchronous, NearSync, and Synchronous replication schedules ensure that an application
and its configuration details synchronize to one or more recovery locations for a smoother
recovery.
• Leap works with sets of physically isolated locations called availability zones. An instance of
Prism Central represents an availability zone.
• One availability zone serves as the primary site for an application while one or more paired
availability zones serve as the recovery sites.
• When paired, the primary site replicates the entities (protection policies, recovery plans, and
recovery points) to recovery sites in the specified time intervals (RPO).
• Entities are replicated back to the primary site when the primary site is up and running to
ensure application high availability.
• The entities you create, or update synchronize continuously between the primary and
recovery sites which enables you to create or update entities at either the primary or the
recovery sites.
• It is possible for one primary site to replicate to and synchronize with:
▪ Up to 2 on-prem recovery clusters in the same physical location as the primary site.
▪ 2 on-prem recovery sites in different physical locations.
▪ 1 on-prem recovery site in a different physical location and Xi Cloud Services.
▪ Xi Cloud Services only.
- Leap with two Prism Centrals (in the same or different availability zones) replicates the data and
protect VMs in the following three schedules:
• Asynchronous (1 hour or greater RPO).
• NearSync (1 to 15 minute RPO).
• Synchronous (0 RPO).
- When performing DR to Nutanix clusters using Leap, you must consider the below:
• You must first enable Leap on both the primary and the secondary sites.
▪ If Leap is not enabled, you can configure protection policies and recovery plans that
synchronize to the paired sites, but you cannot perform failover and failback operations.
▪ If your primary and secondary clusters are at the same site and managed by a single
Prism Central, Leap only needs to be enabled once to apply to both clusters.
• Prism Element that is hosting Prism Central needs to be registered with Prism Central.
• The iSCSI data services IP must be configured on the Prism Element hosting Prism Central,
and must be reachable on port 3260.
- To replicate entities such as protection policies, recovery plans, and recovery points between
different on-prem availability zones, those zones must first be paired with each other.
• If on-prem zones in two different site, managed by different Prism Centrals, are not paired
with each other, you cannot perform DR.
• If the availability zones are at the same site and registered to a single Prism Central, pairing
is not necessary.

36
 Module (08)
Configuring & Managing Cluster Storage
❖ AOS Storage Overview:
- AOS Distributed Storage simplifies storage and data management for virtual environments by
pooling flash and hard disk drive storage across a Nutanix cluster and exporting it as a data store
to the virtualization layer as iSCSI, NFS, and SMB shares, AOS eliminates the need for SAN and NAS
solutions.
- It presents all the storage devices in the cluster to the hypervisor as a pool of storage that
provides cluster-wide storage services, such as snapshots, clones, HA, DR, deduplication,
compression, and erasure coding.
- The Controller VMs (CVMs) running on each node combine to form an interconnected network
within the cluster, where every node in the cluster has access to data from shared SSD, HDD, and
cloud resources.
- AOS Distributed Storage has several capabilities that improve performance:
• Intelligent Tiering:
▪ It continually monitors data access patterns and optimizes data placement on the SSD or
HDD tier, achieving the best performance without administrator intervention.
▪ The SSD tier provides maximum performance for hot data and random I/O, while the
HDD tier provides maximum capacity and economy for cold data and sequential I/O.
• Data Locality:
▪ It ensures that as much of a VM’s data as possible is stored on the node where the VM is
running, and this negates the need for read I/O to go through the network.
▪ Keeping data local optimizes performance and minimizes network congestion.
▪ Every VM’s data is served locally from the CVM and stored preferentially on local storage.
▪ When a VM is moved from one node to another, the migrated VM’s data automatically
follows the VM in the background based on read patterns.
• Automatic Disk Balancing:
▪ It is capable of responding to different workloads and allows different node types (for
example, compute-heavy or storage-heavy nodes) to be mixed in a single cluster.
▪ The native disk balancing feature ensures that data is distributed uniformly among nodes
once storage utilization on a node crosses a particular threshold.
▪ Movement of data is always done within the same tier (that is, SSD or HDD) when disk
balancing is performed. Disk balancing will not move data between storage tiers.
• Data Path Redundancy:
▪ It ensures high availability in the event a Nutanix CVM becomes unavailable for any
reason, Nutanix CVM autopathing automatically reroutes requests to a “healthy” CVM on
another node.

37
 ▪ This failover is fully transparent to the hypervisor and applications. This redirection
continues until the local CVM failure issue is resolved.

❖ Nutanix Storage Terms:

- Storage Pool:
• A storage pool is a group of physical storage devices including PCIe SSD, SSD, and HDD
devices for the cluster.
• The storage pool can span multiple Nutanix nodes and is expanded as the cluster scales. In
most configurations, only a single storage pool is leveraged.
- Storage Container:
• A container is a logical segmentation of the Storage Pool and contains a group of VM or
files (vDisks).
• Some configuration options (e.g., RF) are configured at the container level however, are
applied at the individual VM/file level.
• Containers typically have a 1 to 1 mapping with a datastore (in the case of NFS/SMB).
- vDisk:
• A vDisk is a subset of available storage within a storage container that provides storage to
virtual machines.
• A vDisk is any file over 512 KB on DSF, including VMDKs and VM disks.
• vDisks are broken up into extents, which are grouped and stored on physical disk as an
extent group.
- Volume Group:
• A volume group is a collection of logically related virtual disks or volumes.
• It is attached to one or more VMs or other iSCSI initiators that share the disks in the volume
group.
• You can manage volume groups as a single unit.
• Each volume group contains a UUID, a name, and iSCSI target name.
• You can include volume groups in protection domains configured for asynchronous data
replication exclusively or with VMs, but it cannot be included in a protection domain
configured for Metro Availability, in a protected VStore, or in a consistency group for which
application consistent snapshotting is enabled.
- vBlock:
• A vBlock is a 1MB chunk of virtual address space composing a vDisk.
• For example, a vDisk of 100MB will have 100 x 1MB vBlocks, vBlock 0 would be for 0-1MB,
vBlock 1 would be from 1-2MB, and so forth.
• These vBlocks map to extents which are stored as files on disk as extent groups.

38
- Extent:
• An extent is a 1MB piece of logically contiguous data which consists of N number of
contiguous blocks (varies depending on guest OS block size).
• Extents are modified on a sub-extent basis (aka slice) for granularity and efficiency.
- Extent Group:
• An extent group is a 1MB or 4MB piece of physically contiguous stored data.
• This data is stored as a file on the storage device owned by the CVM.
• Extents are dynamically distributed among extent groups to provide data striping across
nodes/disks to improve performance.
- OpLog:
• The OpLog is similar to a filesystem journal and is built as a staging area to handle bursts of
random writes, coalesce them, and then sequentially drain the data to the extent store.
• For sequential workloads, the OpLog is bypassed and writes go directly to the extent store.
• If data is currently sitting in the OpLog and has not been drained, all read requests will be
directly fulfilled from the OpLog until they have been drained, where they would then be
served by the extent store/unified cache.
- Extent Store:
• The Extent Store is the persistent bulk storage of DSF and spans all device tiers (Optane SSD,
PCIe SSD, SATA SSD, HDD) and is extensible to facilitate additional devices/tiers.
• Data entering the extent store is being drained from the OpLog or is sequential/sustained in
nature and has bypassed the OpLog directly.
• Nutanix Intelligent Lifecycle Manager (ILM) will determine tier placement dynamically based
upon I/O patterns, number of accesses of data and weight given to individual tiers and will
move data between tiers.
- Autonomous Extent Store:
• The Autonomous Extent Store (AES) is a different method for writing/storing data in the
Extent Store.
• It leverages a mix of primarily local + global metadata allowing for much more efficient
sustained performance due to metadata locality.
• For sustained random write workloads, these will bypass the OpLog and be written directly
to the Extent Store using AES.
• For bursty random workloads these will take the typical OpLog I/O path then drain to the
Extent Store using AES where possible.
• As of AOS 5.20, AES is enabled by default for new containers on All Flash Clusters and as of
AOS 6.1 if requirements are met, AES is enabled on new containers created on Hybrid
(SSD+HDD) Clusters.
- Unified Cache:
• The Unified Cache is a read cache which is used for data, metadata and deduplication, and is
stored in the CVM’s memory.

39
 • Upon a read request of data not in the cache, the data will be read from the extent store and
will also be placed into the single-touch pool of the Unified Cache which completely sits in
memory, where it will use LRU (least recently used) until it is evicted from the cache.
• Any subsequent read request will “move” the data into the multi-touch pool.
• Any read request for data in the multi-touch pool will cause the data to go to the peak of
the multi-touch pool where it will be given a new LRU counter.

❖ Replication Factor:
- Replication factor refers to the number of copies of data and metadata that will be maintained on
a cluster.
• A replication factor of 2 means that 2 copies of data will be available (1 original and 1 copy).
• A replication factor 3 means that 3 copies of data will be available (1 original and 2 copies).
- While replication factor 1 is available (only the original data will be maintained, with no copies) it
is not recommended unless your cluster is running applications that provide their own data
protection or high availability.
- How Replication Factor Works:
• The OpLog acts as a staging area to absorb incoming writes onto the low-latency SSD tier.
• When data is written to a local OpLog, it is synchronously replicated to another one or two
Nutanix CVM’s OpLog (one other OpLog for RF2 and two other OpLogs for RF3) before
being acknowledged as a successful write to the host.
• This ensures that the data exists in at least two or three independent locations and is fault
tolerant.
- The replication factor is handled differently for data and metadata.
• For data RF2, there will be 2 copies of data and 3 copies of metadata.
• For data RF3, there will be 3 copies of data and 5 copies of metadata.
- Metadata replication factor cannot be set or configured independently of data replication factor,
and is dependent on data replication factor and the cluster's redundancy factor.
iNote:
* You can view and configure replication factor from both Prism Central and Prism Element
* While redundancy factor is defined and set at the cluster level, replication factor is set at the
storage container level.
- The replication factor and redundancy factor features are very closely connected.
• Redundancy factor 2 supports replication factor 2 only.
• Redundancy factor 3 supports replication factor 2 and 3.

❖ Redundancy Factor:
- Redundancy factor refers to the number of failures (such as a node failure or a disk failure) that a
cluster is able to withstand while still continuing to operate.

40
- By default, Nutanix clusters (with the recommended minimum of 3 nodes) have a redundancy
factor of 2. This means they can tolerate the failure of a single drive or node.
- Enabling redundancy factor 3 allows a cluster to withstand the failure of 2 nodes or disks but it is
important to note that:
• A cluster must have at least 5 nodes, blocks, racks for redundancy factor 3 to be enabled.
• For guest VMs to tolerate the simultaneous failure of 2 nodes or drives in different blocks,
the data must be stored on storage containers with replication factor 3.
iNote:
* You can view and configure redundancy factor from Prism Element only.
* Redundancy Factor cannot be reverted, and you cannot reduce the redundancy factor.

❖ Drive Failure Handling:

- Drives in a Nutanix node store four primary types of data: persistent data (hot-tier and cold- tier);
storage metadata; OpLog & CVM boot files.
- Cold-tier persistent data is stored on the capacity tier of the node. Storage metadata, OpLog, hot-
tier persistent data, and CVM boot files are stored in the performance tier.
- Boot Drive Failure:
• Each Nutanix CVM boots from a SATA-SSD. During cluster operation, this drive also holds
component logs and related files.
• A boot drive failure will eventually cause the associated CVM to fail. The host does not
access the boot drive directly, so other guest VMs can continue to run. Data Path
Redundancy redirects the storage path to another CVM.
• The CVM might restart under certain rare conditions on dual SSD nodes if a boot drive fails,
or if you unmount a boot drive without marking the drive for removal and the data has not
successfully migrated.
- Metadata Drive Failure:
• Cassandra uses up to 4 SSD disks to store the database providing read and write access for
cluster metadata.
• When a metadata drive fails, the local Cassandra process will no longer be able to access its
share of the database and will begin a persistent cycle of restarts until its data is available.
• If Cassandra cannot restart, the Stargate process on that CVM will crash as well. Failure of
both processes results in automatic IO redirection using data path redundancy.
• During the switching process, the host with the failed SSD may report that the shared
storage is unavailable. Guest VM IO on this host will pause until the storage path is restored.
• After redirection occurs, VMs can resume read and write I/O. Performance may decrease
slightly, because the I/O is traveling across the network rather than across the internal
network.
• Multiple drive failures in a single selected domain (node, block, or rack) are also tolerated.

41
 • The Controller VM restarts if a metadata drive fails, or if you remove a metadata drive
without marking the drive for removal and the data has not successfully migrated.
• If Cassandra remains in a failed state for more than 30 minutes, the surviving Cassandra
nodes detach the failed node from the Cassandra database so that the unavailable metadata
can be replicated to the remaining cluster nodes.
• If the Cassandra process restarts and remains running for 5 minutes, the procedure to
detach the node is canceled.
• If the process resumes and is stable after the healing procedure is complete, the node will
be automatically added back to the ring. A node can be manually added to the database
using the nCLI command.
- Data Drive Failure:
• Each node contributes its local storage devices to the cluster storage pool. Cold-tier data is
stored in HDDs, while hot-tier data is stored in SSDs for faster performance.
• Data is replicated across the cluster, so a single data drive failure does not result in data loss.
Nodes containing only SSD drives only have a hot tier.
• When a data drive (HDD/SSD) fails, the cluster receives an alert from the host and
immediately begins working to create a second replica of any guest VM data that was stored
on the drive.
• For a brief period of time, guest VMs with files on the failed data drive will need to read
across the network.
• In a cluster with a replication factor 2, losing a second drive in a different domain (node,
block, or rack) before the cluster heals can result in some VM data loss to both replicas.

❖ Node Failure Handling:

- A Nutanix node consists of a physical host and a CVM. Either of these components can fail
without impacting the rest of the cluster.
- Controller VM (CVM) Failure:
• In case CVM failure, the storage traffic is served by another CVM in the cluster.
• The hypervisor and CVM communicate using a private network on a dedicated virtual switch
meaning that the entire storage traffic is routed through an internal IP address on the CVM.
▪ The external IP address of the CVM is used for remote replication and for CVM to CVM
communication.
• In the event of a local CVM failure, AOS Distributed Storage automatically detects the
outage and redirects storage traffic to another CVM in the cluster over the network.
• The re-routing is done transparently to the hypervisor and to the VMs running on the host.
So even if a CVM is powered-off, VMs continue to perform I/O operations.
• AOS Distributed Storage is also self-healing, which means that it detects when a CVM is
powered-off and automatically reboots the local CVM.
• Once the local CVM is available, the traffic is seamlessly transferred back to be served by the
local CVM.

42
 • AOS Distributed Storage uses replication factor and checksum to ensure data redundancy
and availability in the case of a node or disk failure or corruption.
- AHV Host Failure:
• If a node fails, all HA-protected VMs can be automatically restarted on other nodes in the
cluster.
▪ Users who are accessing HA-protected VMs will notice that their VMs are unavailable
while they are being restarted on the new host.
▪ Any VMs that are not protected by HA will need to be manually restarted.
• Two Acropolis Services, Curator and Stargate, respond to two types of issues that arise from
host failure:
▪ First, when the guest VM begins reading across the network, Stargate begins migrating
those extents to the new host. This improves performance for the guest VM.
▪ Second, Curator will notice that there is a missing replica of those extents, and instruct
Stargate to begin creating a second replica.
• In a loaded cluster, the bigger concern is additional risk to guest VM data. With two sets of
inaccessible physical disks, there is a chance that some VM data extents will be missing
entirely, and I/O requests will not be served.

❖ Block Fault Tolerance:

- A block is a rack-mountable enclosure that contains 1 to 4 Nutanix nodes. All nodes in a block
shares power supplies, front control panels, backplane, and fans.
- Block fault tolerance allows a Nutanix cluster to make redundant copies of data and metadata
and place the copies on nodes in different blocks.
- With block fault tolerance enabled, guest VMs can continue to run after a block failure because
redundant copies of guest VM data and metadata exist on other blocks.
- Nutanix offers block fault tolerance as:
• Opt-in block fault tolerance: It offers guaranteed data resiliency when required conditions
are met.
• Best-effort fault tolerance: The copies of the data remain on the same block when there is
insufficient space across all blocks.
- Block fault tolerance is applied automatically in the following scenarios:
• Every storage tier in the cluster contains at least 1 drive on each block.
• Every storage container in the cluster has replication factor of at least 2.
• For replication factor 2, there are a minimum of 3 blocks in the cluster.
• If the replication factor of storage containers in the cluster is 2, then at least 2 blocks require
free space.
• If the replication factor is 3, then at least 3 blocks require free space.
• A minimum of 4 blocks for RF2 or 6 blocks for RF3 is required to maintain block awareness
if erasure coding is enabled on any storage container.

43
iNote:
* Since block & rack fault tolerance is configured at a cluster level, it must be accessed from Prism
Element and not Prism Central.

❖ Rack Fault Tolerance:

- Rack fault tolerance allows a cluster to withstand the failure of 1 rack in the case of RF2, or 2
racks in the case of RF3.
- In this scenario, VMs will also continue to run. Rack fault tolerance enables this by making
redundant copies of data and placing them on nodes that are in different racks.
- To enable rack fault tolerance, you must specify the mapping of the blocks to the racks based on
the actual placement of the blocks in the datacenter. The minimum cluster requirements are:
• Replication factor 2 - requires 3 racks (4 with Erasure Coding)
• Replication factor 3 - requires 5 racks (6, with Erasure Coding), 1 node in each rack.

❖ Storage Container Management:

- By default, a storage pool is created when the cluster is configured, along with 3 default Storage
containers:
• NutanixManagementShare: This is a built-in storage container used by Nutanix Files and
Self-Service Portal (SSP) file storage, feature upgrades, and other operations. It is not
intended to be used as storage for vDisks.
• SelfServiceContainer: This is a built-in storage container used by Image Service features.
• Default-container: This built-in container can be used for regular storage.
- In general, when updating a storage container, remember that:
• You cannot rename a storage container through the Update Storage Container dialog box.
• You also cannot rename a storage container if it contains vDisks.
• You cannot change the replication factor when updating a storage container in Prism. This
can be done only via the command-line interface.
• If the compression policy is changed from compressed to uncompressed (or vice versa), the
data in the storage container will be uncompressed or compressed as a background process
when the next data scan detects data that needs this change.

❖ Reserved and Advertised Capacity:

- By default, when a storage container is created, it has access to all of the unused storage in the
storage pool.
- If a storage pool has multiple storage containers, a single storage container may consume all
available space and leave none for the others. Capacity reservation is used to mitigate this potential
problem.

44
- Capacity reservation allows you to reserve a minimum amount of space in a storage pool for a
specific storage container, which prevents this space from being used by other storage containers.
• Reserved Capacity: This is the minimum amount of space that will be reserved on the
cluster for use by this storage container.
• Advertised Capacity: This is the maximum amount of space that this container is allowed to
use.

❖ Understanding Compression:
- Nutanix provides two forms of compression:
• Inline Compression:
▪ It is enabled by default when you create a new storage container.
▪ Inline compression for large I/O size or sequential data streams compresses the data in
memory prior to writing it out to disk.
▪ The data is written uncompressed to the OpLog then, it is coalesced, compressed in
memory and written out to the extent store.
• Post-process Compression:
▪ It sees the data written uncompressed to disk before the Nutanix MapReduce framework
compresses the data cluster-wide.
▪ 60 minutes is also the Nutanix-recommended delay for post process compression.

❖ Redundancy Factor:
- Nutanix provides two forms of deduplication:
• Cache Deduplication: It means deduplication performed on data in memory. It is applied to
the read cache to optimize performance.
• Capacity Deduplication: It means deduplication performed on the data in hard disk storage
(HDD). Note that you can enable capacity deduplication only if cache deduplication is
enabled, and if you have a Pro or higher license.
- To enable deduplication, the CVMs in the cluster needs to be configured with additional RAM.
Each CVM in a cluster needs 24 GB of RAM for cache deduplication, and 32 GB of RAM for capacity
deduplication.
- It is recommended to enable deduplication for full clones, physical-to-virtual migration, and
persistent desktops.
- It is not recommended to enable deduplication for the following workloads:
• Linked clones or Nutanix VAAI clones: Duplicate data is managed efficiently by DSF so,
deduplication has no additional benefit.
• Server workloads: Redundant data is minimal so there may be no significant benefit from
deduplication.

45
 ❖ Erasure Coding Overview:
- When we change the replication factor from 2 to 3 to handle additional failures, this increase,
reduces usable disk space by creating additional redundant copies of the data.
- Nutanix addresses this drawback with the Erasure Coding (EC-X) feature, which increases usable
disk space while maintaining the same cluster resiliency by striping individual data blocks and
associated parity blocks across nodes rather than disks.
- In the event of a failure, the system uses the parity block along with the remaining blocks in the
erasure strip to recalculate the missing data onto a new node.
- All blocks associated with erasure coding strips are stored on separate nodes. Each node can then
take part in subsequent rebuilds, which reduces potential rebuild time.
- EC-X works best on cold data, archives, and backups. Containers with applications that incur
numerous overwrites, such as log file analysis or sensor data, require a longer delay than the one-
hour EC-X post-processing default.
- How Erasure Coding Works:
• Consider a 6-node cluster with redundancy factor 2, that contains four data blocks (a, b, c,
and d).
▪ The black text in the following figure represents the data blocks and the orange text
represents the copies.

• When the data becomes cold, the erasure coding engine calculates the value of parity (P) by
performing an exclusive OR operation. Once parity has been calculated, the copies are
removed and replaced by the parity information as shown in the following figure.

• In the event of a failure, the missing data block is rebuilt using the rest of the erasure coded
stripe (a b d and P). The restored block (block c, in this example) is then placed on a node
that does not have any other members of this erasure coded stripe.

46
- When a cluster is configured for the redundancy factor 3, two parity blocks are maintained so that
the erasure coded data has the same resiliency as the replicated data. An erasure coded stripe with
two parity blocks can withstand the failure of two nodes.
- A minimum of 4 nodes are needed to enable erasure coding in a cluster with RF2, while at least 6
nodes are needed in a cluster with RF3. If the number of nodes in the cluster is fewer than the
required number, erasure coding cannot be enabled.

47
 Module (09)
Migrating Workloads with Nutanix Move
❖ Nutanix Move Overview:
- Nutanix Move is a migration tool that can be used to quickly and easily transfers VMs from ESXi,
Hyper-V, and public cloud environments to Nutanix AHV. It is a free product that can be deployed
as a VM in an AHV environment and is managed using a HTML5-based interface.
- Move supports migration from the following sources to targets:
• VMware ESXi to Nutanix Clusters on AWS
• VMware ESXi to AHV
• VMware ESXi on legacy infrastructure to VMware ESXi on Nutanix
• Microsoft Hyper-V to Nutanix Clusters on AWS
• Microsoft Hyper-V to AHV
• Microsoft Hyper-V to VMware ESXi on Nutanix
• AWS EC2 to Nutanix Clusters on AWS
• AWS EC2 to AHV
• AWS EC2 to VMware ESXi on Nutanix
• Nutanix AHV to AWS EC2
• Microsoft Azure to AHV
• Microsoft Azure to VMware ESXi on Nutanix
- The Nutanix Move VM is typically hosted on the target AHV cluster and its software services can
be grouped into three major components.
• Management Server:
▪ It maintains source and target cluster information, as well as migration plan details and
current status.
▪ It also allows APIs and the UI to create and manage migration plans.
• Agents for Source & Target:
▪ The source agent is a platform-specific (ESXi, Hyper-V, or cloud) software component
that schedules migration copy requests through disk readers.
▪ It collects source cluster and VM details and helps the user select the VMs to migrate
using the management server UI.
▪ The target agent collects and keeps inventory information for the target cluster, allowing
you to create migration plans.
▪ It also mounts the container in the target to prepare the disk writer to copy the data.
▪ At cutover, the target agent converts disk formats to support AHV.
• Disk Readers and Writers:
▪ Disk reader processes use source-specific APIs to read data and coordinate with disk
writer processes to complete outstanding copy operations.

48
 ▪ The disk reader checkpoints copy operations to handle any failures and resume
operations as needed.
- To install Nutanix Move on AHV:
• Download Nutanix Move from Nutanix Support Portal.
o Entities > Downloads > Cloud category
• Create a new VM:
o At least 2 cores, 2 vCPUs per core, and 8GB of memory.
• Upload the Move QCOW2 image to the Image Service through Prism.
• Mount the QCOW2 image so that you can log into the Move UI.
• After the Move VM has been created and powered on, you can log into the Move UI, access
the Move dashboard, and perform VM migrations.
- Move can be upgraded from within the Move VM if a connection to the internet is available or if
you have downloaded the required binary.
- Move has a collection of real-time logs that you can view at any time to investigate issues with a
VM or a migration.
• The logs bundle includes Move logs, agent logs in the case of Hyper-V, and ping statistics
for source ESXi and target AHV hosts.

❖ VM Migration Nutanix Move Overview:

- Move does not currently support all hypervisor environments as migration sources or non-AHV
environments as targets, so heterogeneous source and target environments currently require other
migration methods.
- The overall migration workflow using Nutanix Move is as follows:
• Log in to Move.
• Add environment for migration.
• Create a migration plan.
• Prepare VMs either automatically or manually.
• Perform a test migration or migrate directly to your target environment.
• Manage your migration plans.
- When performing a migration:
• Move checks to ensure that the target environment has enough compute and storage
resources to support the VMs added to a migration plan.
• Move can provide a summary to indicate why you cannot migrate certain VMs (for example,
if the VM does not have VMware tools installed or meet virtual hardware version level
minimums).
- For VMware ESXi sources:
• Move uses VMware vSphere Storage APIs: Data Protection (VADP) to manage the replication
process, so you do not need to have agents installed in the VMs or the ESXi hosts.

49
• You have the option to allow Move to connect to VMs directly in order to install the VirtIO
drivers compatible with AHV and to capture network settings to carry over to the target
environment.
• You can specify the credentials to connect to the VMs selected in a plan either for all VMs at
once or individually as needed.
• You can specify network mappings to match the source and destination networks for the
VMs.
• This migration process involves creating ESXi-based snapshots for each VM, then replicating
the virtual disks to the specified AHV container.
• Move stores the VMDK files for the migrating VMs in a temporary folder and incrementally
uses changed-block tracking (CBT) APIs and continued snapshot operations to keep them
up to date.
• When it is time to cut over and complete the migration, Move powers off the source VMs
and disconnects the vNICs.
• Once all data replication is complete, Move uses the AHV image service to convert the
VMDK files to the native RAW format used by AHV.
▪ Because the disk formats are the same, conversion from VMDK to RAW is extremely fast
limiting downtime.
• To complete the migration, Move powers the VMs on in the target environment and
removes all temporary VMDK files and converted images in the AHV image service.

50
 Module (10)
Monitoring Cluster Performance
❖ The Analysis Dashboard:
- The Analysis dashboard allows you to create custom charts or use built-in ones to analyze various
aspects of cluster performance.
- When you open the Analysis dashboard, it will display the last session that you were working on.
• If you are accessing the Analysis dashboard for the first time, you will see a default, system-
generated session, named Analysis Session.
- A session is simply a grouping of charts and metrics allow you examine different infrastructure
and entity metrics without having to create new or custom charts multiple times.
• The default, system-generated session does not contain any performance data. However, it
displays a chart of alerts and events over time called the Alerts and Events monitor.

- On the Analysis dashboard, you can perform a number of different actions:

• Add custom charts.
• View a list of unresolved alerts.
• Customize the date range for which information is displayed in charts.
- Clicking the Alerts and Events link at the top right of the page will display a pane with two tabs:
• Alerts tab:
▪ It displays a number of critical & warning alerts, color coded in red & yellow respectively.
▪ These alerts are categorized by Entity Type (Prism Central VM, Cluster, Host, VM, Remote
Site, and Volume Group), but you can sort the list by either Impact or Severity.
• Events tab:
▪ It displays the total number of cluster events.
▪ These events are displayed in three categories: User Action, Behavioral Anomaly and
System Action.

51
 ❖ The Session Dashboard:
- The Sessions dashboard allows you to create custom groupings of charts and data, and switch
between them quickly and easily for analysis.
- Sessions allow you to customize and organize your infrastructure data, and view information in a
structured format that is comfortable for your specific monitoring needs.

- The Sessions dashboard displays a list of both system-defined and custom sessions, and includes
a number of options for filtering, creating, and managing sessions.
• Only one system-defined session is available by default: the Analysis session.
• Only custom sessions can be deleted; system-defined sessions cannot be deleted. However,
both system-defined and custom sessions can be edited.
• For step-by-step instructions, see the Creating a New Session section.

❖ Understanding Charts:
- In both Prism Central and Prism Element, you can create two types of charts:
• Metric Chart:
▪ It allows you to monitor the performance of a single metric for one or more entities of
the same entity type.
▪ For example, if you were to create a single chart that monitors CPU utilization for all
CVMs in a cluster, that would be a metric chart.
• Entity Chart:
▪ It allows you to monitor multiple metrics for a single entity.
▪ For example, if you were to create a single chart that monitored the CPU and memory for
a single VM, that would be an entity chart.

❖ The Reports Dashboard:

- The Reports dashboard allows to create, customize, and export reports about the infrastructure
resources and deliver them directly to a mailbox if needed.

52
- The Reports dashboard displays a list of all system-defined and custom reports. Two reports are
available out-of-the-box: Environment Summary and Cluster Efficiency Summary and these can be
identified by the System tag next to their names.
• For step-by-step instructions, see the Creating a New Report & Configuring Report Settings
sections.

❖ Main Dashboard in Prism Central:

- To get performance data at a glance, Prism Central allows you to both customize the main
dashboard and create custom dashboards.
- Prism Central's main dashboard comes with a collection of out-of-the-box widgets that provide
insight into the health, performance, and overall status of your cluster.
- This dashboard is highly customizable; Widgets can be added, removed, or rearranged to make
infrastructure monitoring both simple and comfortable.

53
 Module (11)
Monitoring Cluster Health
❖ Health Monitoring in Prism Central:
- Main & Custom Dashboards:
• Health information at a glance can be obtained from the main dashboard, which includes an
Alerts widget and a VM efficiency widget.
• Additionally, since you can customize the main dashboard or create your own dashboard,
you can also add widgets that provide additional health information.
- Events & Behavioral Anomalies:
• Each of the numbers in the widget is clickable so, if we wanted to investigate the anomalies
in more detail, we need to click the number in the widget.
• This will display a filtered version of the Events dashboard (with filters for event type, time,
and cluster), with a list of behavioral anomalies that we can review and remediate as needed.
- Search Preview & Details:
• To access summarized health information for all entities in all clusters managed by Prism
Central, in the search field, type health and press Enter.
• Prism Central will display a page with a list of all entities, their name, health status (good,
warning, critical, unknown), latency information, and critical and alert warning counts.
• However, the Search results page only contains summarized information. If we want details,
we need to use the View All links at the bottom of each group of information.

❖ Home Dashboard in Prism Element:

- The Health Widget:
• It displays the health status for the cluster as a whole, as well as cluster entities.
• The three possible health states are good, warning, and critical.
• Scrolling down in the widget will display the health status of hosts, VMs, cluster services,
disks, storage containers, and storage pools.
• Clicking the widget will take you to the Health dashboard, where you can view more details
about each or all of these entities.
- The Data Resiliency Status Widget:
• It summarizes the number of failures that a cluster can withstand.
• Moving your mouse cursor over the question marks next to Failure Domain and Fault
Tolerance will display a small popup with more information about each item.
- The Alerts and Events Widgets:
• There are four different alerts and events widgets in the Home dashboard.
• The Alerts widgets display the most recent unresolved alerts on the cluster.

54
 • The Events widget displays the total number of events that have occurred on the cluster, as
well as how recently the last event occurred.

❖ Nutanix Cluster Check:

- Nutanix Cluster Check (NCC) is software that can help diagnose cluster health and identify
configurations qualified and recommended by Nutanix. Depending on the issue discovered, NCC
raises an alert or automatically creates Nutanix Support cases.
iNote:
* In Prism Element, NCC can be run from both the web console and the command line. However, in
Prism Central, NCC can be run from the command line only.
- NCC tests can complete with one of the following status types:
• PASS:
▪ This means that the tested aspect of the cluster is healthy, and no action is required.
▪ A check can also return a PASS status if it is not applicable.
• FAIL:
▪ This means that the tested aspect of the cluster is not healthy and must be addressed.
▪ This message requires an immediate action.
• WARN:
▪ This means that the plugin returned an unexpected value that you must investigate.
▪ This message requires user intervention which you should resolve as soon as possible to
help maintain cluster heath.
• INFO:
▪ This means that the plugin returned an expected value that however cannot be evaluated
as PASS/FAIL.
▪ In some cases, the message might indicate a recommendation from Nutanix.
• ERR:
▪ This means that the plugin failed to execute.
▪ This message represents an error with the check execution and not necessarily an error
with the cluster entity.
- NCC health checks can also be run manually from the Prism Element web console. Moreover,
Prism Element allows you to configure a custom schedule for NCC.

❖ Nutanix Common Log Files:

- Nutanix Logs Root:
• The /home/nutanix/data/logs directory stores Nutanix logs.
• This location contains all the Nutanix process logs at the INFO, WARNING, ERROR and
FATAL levels.
• It contains the directories for system stats (sysstats), and Cassandra system logs (cassandra).

55
- Self-Monitoring (sysstats) Logs:
• The /home/nutanix/data/logs/sysstats directory stores the self-monitoring logs.
• The node self-monitors itself by running several Linux tools every few minutes:
▪ The ping command is used to check if a network is available and if a host is reachable.
▪ The iostat command is used to monitor both physical and logical input/output devices
by observing the time for which these devices are active.
▪ The sar command stands for system activity report, and is used to monitor various
aspects of system performance such as CPU & memory usage, disk usage, and so on.
▪ The df command stands for disk filesystem (or disk free) and is used to check the
amount of free and used disk space.
• The results of these checks, and others, are available in the self-monitoring logs.
- Consolidated Audit Logs:
• The data/logs/consolidated_audit.log directory stores the audit logs that allows you to view a
list of actions performed across the clusters.
- Cassandra Logs:
• The /home/nutanix/data/logs/cassandra directory stores the Cassandra metadata DB logs.
• The Nutanix process that starts the Cassandra database (cassandra_monitor) logs to the
/home/nutanix/data/logs directory.
- Genesis Logs:
• Genesis logs are available at /home/nutanix/data/logs/genesis.out.
• When checking the status of the cluster services, if any of the services are down, or the
Controller VM is reporting Down with no process listing, review this log to determine why
the service did not start, or why Genesis is not running properly.
- Prism Gateway Logs:
• The Prism log is located at /home/nutanix/data/logs/prism_gateway.log on the Prism leader.
• The Prism leader is the node which is running the web server for the Nutanix UI.
• This is the log to analyze if there are problems with the UI, such as long loading times.
- Stargate Logs:
• Stargate logs are located at /home/nutanix/data/logs/stargate.
- Zookeeper Logs:
• Zookeeper logs are located at /home/nutanix/data/logs/zookeeper.out which contain the
status of the Zookeeper service.
• If one of the other logs specifies that it is unable to contact Zookeeper and it is affecting
cluster operations, you may want to look at this log to find the error Zookeeper is reporting.
- You can configure syslog monitoring in Prism Central which will forward system logs of the
clusters registered with Prism Central to one or more external syslog server. Also, you can configure
multiple remote syslog servers if needed, and can configure log forwarding so that separate log
modules are sent to different servers.

56
 Module (12)
Investigating & Remediating Performance Issues
❖ The Alerts Dashboard:
- The Alerts dashboard provides a list of alert messages generated for all clusters that are
registered with Prism Central.
- It displays the List tab by default, and an Alert Policies drop-down menu that allows you to
access different views for different types of alert policies.
- You can use the buttons at the top of the page to Acknowledge the alert if you intend to take
action, and to Resolve the alert if the issue has been remediated.
- The Alert Policies tab has three views that you can choose from:
• User Defined view: It displays all custom alert policies that are user defined.
• System Defined view: It displays all default alert policies that are system defined.
• External Defined view: It displays all alert policies that are defined by external entities,
typically an application through an API. You cannot modify an externally defined policy, but
you can view it.
- The Alerts dashboard in Prism Element is similar to the Alerts dashboard in Prism Central, with
few differences. In Prism Element, you can add custom views and apply filters. However, grouping
alerts by cluster, severity, and impact type is unique to Prism Central.
- Another difference is that Prism Element has a single dashboard with multiple tabs for Alerts,
Alert Policies and Events. However, Prism Central has one dedicated dashboards for Alerts, and
another dashboard for Events, with Alert Policies accessible from the Alerts dashboard.
- Prism Central allows you to create custom alert policies, and modify existing, system-defined
policies.
- Auto-resolving an alert will only resolve the alert message itself. Selecting the auto-resolve option
will not enable the cluster to remediate the issue. Resolution may require the creation and
deployment of a playbook or administrator intervention.
- Prism Central allows you to stop one or more alerts from being generated. You can stop a single
alert via the List tab, or stop multiple alerts via the command line. You can stop alerts of only
cluster entity type.
- Stopping Multiple Alerts:
• Log on to Prism Central and create a specifications file.
• Specify the values for the following parameters:
▪ Under scope_entity_list, specify scope of the entities for which you want to stop the
alerts.
▪ Under schedule_list, define the schedule during which the alerts should be stopped.
• Create a configuration to stop the alerts by running this command:

57
 o $ nuclei --username admin --password password blackout.create
spec_file=filepath
• List all the configurations that stop alerts by running this command:
o $ nuclei --username admin --password password blackout.list
▪ The output of this command lists the UUID of each configuration that you have created.
• Update the configuration by running this command:
o $ nuclei --username admin --password password blackout.put uuid
spec_file=filepath
• Finally, to get a configuration, run the following command:
o $ nuclei --username admin --password password blackout.get uuid

- Prism Central allows you to email a notification to one or more recipients whenever an alert is
generated knowing that any emails you configure in Prism Central will be sent in addition to emails
that may have already been configured in Prism Element.
• If you enable alerts through Prism Central and do not want to receive double email
notifications for the same alert, disable customer email notification for those alerts on the
individual clusters through Prism Element.

❖ The Events Dashboard:

- The Events dashboard contains messages that describe cluster-related activities performed by the
user and the system.
- There are three types of events that the system generates:
• User Action:
▪ These types of events, when combined with the Audits feature in Prism Central, provide
you with a detailed view of the activity that occurs on a cluster.
▪ You can very easily identify which entities have been created, changed, or modified, and
which users are responsible for which actions and activity on a cluster.
• Behavioral Anomaly:
▪ It is a sudden unexpected change in some aspect of system or entity performance
related to the network, storage, CPU, memory, and so on.
▪ A behavioral anomaly is a temporary, unexpected increase in performance but not severe
enough or persistent enough to warrant the generation of an alert.
▪ However, repeated occurrences of the same or similar behavioral anomalies on the same
entities are often worth investigating.
▪ Behavioral anomalies are based on deviations from a normal behavior band, which the
system defines for various metrics based on historical data.
▪ The anomaly detection module monitors these predefined metrics on a daily basis and
publishes baseline values for each of them.
▪ The anomaly detection module then measures usage every 5 minutes and compares that
usage with the expected values. If the observed value is outside the band, it flags that
value as an anomaly.

58
 • System Action:
▪ Events created by system action are only informational and there is no user intervention
required.

❖ Investigating VMs Performance Issues:

- To identify inefficient VMs in a cluster, you will need to look at the profile of your VMs. The VM
Efficiency widget lists four types of inefficient VMs:
• Constrained VM:
▪ It does not have enough resources and can lead to performance bottlenecks.
▪ A VM is considered constrained when it exhibits one or more of the following baseline
values, based on the past 21 days:
o CPU usage > 90% (moderate), 95% (severe)
o CPU ready time > 5% , 10%
o Memory usage > 90%, 95%
o Memory swap rate > 0 Kbps (no moderate value)
▪ If a VM is constrained, you can hot-plug CPU and memory on the VM while the VM in
powered on.
• Over-provisioned VM:
▪ It is the opposite of a constrained VM, meaning it is a VM that is over-sized and wasting
resources which are not needed.
▪ A VM is considered over-provisioned when it exhibits one or more of the following
baseline values, based on the past 21 days:
o CPU usage < 50% (moderate) or < 20% (severe) and CPU ready time < 5%
o Memory usage < 50% (moderate) or < 20% (severe) and memory swap = 0 Kbps
▪ If a VM is over-provisioned, you can remove resources from it but the VM must be
powered off if you want to remove resources.
• Bully VM:
▪ It consumes too many resources and causes other VMs to starve.
▪ A VM is considered a bully when it exhibits one or more of the following conditions for
over 1 hour:
o CPU ready time > 5%
o Memory swap rate > 0 kbps
o Host I/O Stargate CPU usage > 85%
• Inactive VM:
▪ Dead VM: A VM is considered dead when it has been powered off for at least 21 days.
▪ Zombie VM: A VM is considered a zombie when it is powered on but does fewer than 30
read or write I/Os (total) and receives or transfers fewer than 1000 bytes per day for the
past 21 days.

59
 Module (13)
Performing Cluster Maintenance
❖ Performing a Cluster Health Check:
- Nutanix recommends using Prism web console or REST API to perform most administrative
functions of a Nutanix cluster whenever possible and disabling CVM SSH access with password or
key authentication.
- Before you perform operations such as restarting a CVM, restarting an AHV host, or putting an
AHV host into maintenance mode, you will need to perform cluster health check to determine if
the cluster can tolerate a single-node failure.
- If you receive alerts indicating expired encryption certificates or a key manager is not reachable,
resolve these issues before you shut down the cluster. If you do not resolve these issues, data loss
of the cluster might occur.
- You can log on to Controller VM (CVM) with SSH and check the fault tolerance status of the
cluster by running this command: $ ncli cluster get-domain-fault-tolerance-status
type=node

• The value of the Current Fault Tolerance row in the output must be at least 1 for all the
nodes in the cluster.

❖ Node Maintenance:
- You can only place 1 node at a time in maintenance mode for each cluster.
• When a host is in maintenance mode, the CVM is placed in maintenance mode as part of
the node maintenance operation and any associated RF1 VMs are powered off.
• The cluster marks the host as unschedulable so that no new VM instances are created on it.
- When a node is placed in the maintenance mode from the Prism web console, an attempt is
made to evacuate VMs from the host.
• If the evacuation attempt fails, the host remains in the "entering maintenance mode" state,
where it is marked unschedulable, waiting for user remediation.
- Non-migratable VMs are powered-off while live migratable or high availability (HA) VMs are
moved from the original host to other hosts in the cluster.
• After exiting maintenance mode, all non-migratable guest VMs are powered on again and
live migrated VMs are automatically restored on their original host.
• VMs with CPU passthrough or PCI passthrough, pinned VMs (with host affinity policies), and
RF1 VMs are not migrated to other AHV hosts in the cluster when a node undergoes
maintenance.
- When a node enters the maintenance mode, the high-level tasks are performed internally:
• The AHV host initiates entering the maintenance mode.

60
 • HA VMs are live migrated.
• Pinned and RF1 VMs are powered-off.
• The AHV host completes entering the maintenance mode.
• The CVM enters the maintenance mode.
• The CVM is shutdown.
- As the node exits maintenance mode, the following high-level tasks are performed internally:
• The CVM is powered on.
• The CVM is taken out of maintenance.
• The host is taken out of maintenance.
• RF1 VMs continue to be powered on and VMs migrate to restore host locality.

❖ Starting/Stopping Nodes & Clusters:

- Before you begin a node shutdown process, you will need to:
• Check if the cluster can tolerate a single-node failure.
▪ Do not proceed if the cluster cannot tolerate a single-node failure.
• Put the node you want to shut down into maintenance mode.
• Verify the data resiliency status of your cluster.
▪ If the cluster has replication factor 2, you can only shut down 1 node for each cluster.
▪ If an RF2 cluster would have more than one node shut down, shut down the cluster.
- To shut down a node:
• Log on to the Controller VM (CVM) on the host you want to shut down using SSH and run
this command: $ cvm_shutdown -P now
• Log on to the AHV host with SSH: # shutdown -h now
- To start a node in a cluster, turn on the node on the hardware device.
• The CVM will start automatically when you reboot the node.
• If the node is in maintenance mode, use SSH to log on to the CVM and remove the node
from maintenance mode.
• Next, log on to another CVM in the Nutanix cluster with SSH and verify that all services on
all the CVMs are running by running this command: $ cluster status
- Before you shut down a cluster, you will need to:
• Upgrade to the most recent version of NCC.
• Log on to a Controller VM (CVM) with SSH and run a complete NCC health check using this
command: $ ncc health_checks run_all
▪ If you receive any failure or error messages, resolve those issues by referring to the KB
articles indicated in the output of the NCC check results. I
▪ If you are unable to resolve these issues, contact Nutanix Support.
- To shut down an AHV cluster:
• Shut down the services or VMs associated with the AOS features or Nutanix products.

61
 • Shut down all guest VMs in the cluster using Prism Element.
▪ If the cluster has a large number of VMs, shut down them using Prism Central.
▪ You can also use acli to shut down a large number of running VMs.
• Stop the Nutanix cluster and shut down all the CVMs in the cluster.
• After all the CVMs are shut down, shut down each node in the cluster.
• Perform the maintenance activity and start all the nodes and the cluster.

❖ Cluster Modification:
- The cluster expansion process compares the AOS version on the existing and new nodes and
performs any upgrades necessary for all nodes to have the same AOS version.
- Before you add one or mode nodes to an existing cluster, you will need to:
• Review the relevant sections in Prerequisites & Requirements before attempting to add a
node to the cluster.
• Check the Health Dashboard to see if any health checks are failing.
• Allow any current expand cluster operations to complete.
• Ensure that all nodes are in the correct metadata state by checking the Hardware dashboard.
- When expanding a cluster, some hosts cannot be automatically discovered and require manual
entry of hypervisor IP.
- Before you remove a node, consider the following:
• The Prism web console displays a warning message that you need to reclaim the license
after you have removed the node.
• Removing a node takes some time because data on that node must be migrated to other
nodes before it can be removed from the cluster.
• (Hyper-V only) Initiating a removal of a node running Hyper-V fails if the node is running as
a part of a Hyper-V failover cluster.
• (ESXi only) you must use the management tools provided by VMware to first migrate all the
VMs off the host, disconnect it from vCenter, then remove it from the vCenter cluster.
- If the node that you are trying to remove is unreachable or powered off, a notification is triggered
in the Prism Element UI alerting you that the storage utilization for this node could not be
calculated and also suggesting the possible impact of removing this node.

62