Mastering ISE Upgrades

Best Practices, Tips and Tricks

Romain PASSEREL, Security Consulting Engineer

BRKSEC-2889
 • Glossary & Reminders
• Why Upgrading ISE?
• Preparing the Upgrade

Agenda • Performing the Upgrade

• Conclusion

BRKSEC-2889 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
About Romain PASSEREL
#whoami
• Security Consulting Engineer
• Joined Cisco in September 2020 (Graduate Program)
• TAC rotation (4 months) in Krakow ISE Team

• Providing Security Professional Services (PS) for CX

• Specialized on ISE, Secure Firewall (FMC, FTD), ASA and Secure Client
• Experience in automation and cloud services (Umbrella, Duo, ..)

• Working on the Paris 2024 Olympic Project

• Fan of music and aviation!

BRKSEC-2889 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Glossary &
Reminders
Glossary For Reference

• ISE – Identity Service Engine ISE • URT – Upgrade Readiness Tool

• PAN – Policy Administration
Node (Configuration) • EOL – End Of Life
• PPAN – Primary PAN • VM – Virtual Machine
• SPAN – Secondary PAN • GUI – Graphical User Interface
• CLI – Command Line Interface
• MNT – Monitoring Node
• AD – Active Directory
(Operational Data)
• MDM – Mobile Device Management
• PMNT – Primary MNT
• AWS – Amazon Web Services
• SMNT – Secondary MNT
• OCI – Oracle Cloud Infrastructure
• PSN – Policy Service Node
• SNS – Secure Network Server

BRKSEC-2889 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
Reminders For Reference
Types of Deployments

Evaluation Medium Deployment

Standalone 2 x PAN + MNT
PAN + MNT + PSN 5 PSN (6 since 3.0)

Large Deployment
Small Deployment 2 x PAN
2(.5) nodes 2 x MNT
PAN + MNT + PSN 50 PSN

Performance and Scalability Guide for Cisco Identity Services Engine - Cisco

BRKSEC-2889 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
Reminders For Reference
ISE platforms
SNS 3515 EOL
SNS 3595
SNS 3615
SNS 3655 EOL
SNS 3695
SNS 3715
SNS 3755
SNS 3795
Traditional VM
AWS
Azure & OCI

2.7 3.0 3.1 3.2 3.3

BRKSEC-2889 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
Best Practices, Tips & Tricks

This green medal icon will indicate some best

practices.

The blue lightbulb is gathering Tips !

And finally, this unknown non-malicious hacker will give you

some Tricks.

BRKSEC-2889 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
Why Upgrading
ISE ?
ISE Lifecycle Software Lifecycle Support Statement -
ISE - Cisco

• Since ISE 2.7, no more long/short term release

• All versions are entitled to the same lifecycle
• Plan to release a new version every 8 months
12M 6M 12M 12M
FCS EOL SW EoSW LDoS
2.7 28M 6M 12M 12M
3.0 28M 6M 12M 12M
3.1
3.2
2019------2020------2021------2022------2023------2024------2025

BRKSEC-2889 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
 • Enhance Product Stability

• Fix Security Vulnerabilities

Reasons to Upgrade • Integration with other

solutions

• and…

BRKSEC-2889 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
New Features ! For Reference New

• ISE 3.1 new features : • ISE 3.2 new features : • ISE 3.3 new features :
• Release Notes for Cisco • Release Notes for Cisco • Release Notes for Cisco
Identity Services Engine, Identity Services Engine, Identity Services Engine,
Release 3.1 – Cisco Release 3.2 – Cisco Release 3.3 – Cisco

• API enhancements • Data Connect • Certificate based API

Better Posture calls
•
• Better automation
Better logging and • AI powered profiling
•
• Cloud support
alarms • Native IPSec
• Dark mode
• New upgrade method • New split-upgrade
workflow

BRKSEC-2889 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
Preparing the
Upgrade
Upgrading ISE is not
easy, unless you are
well prepared!

Pre-flight Checklist | David

© 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public
Cisco Official Documentation

• Cisco Identity Services Engine Upgrade Journey, Release 3.2 - Cisco

BRKSEC-2889 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
 • Check the suggested release
(golden star)
• Check compatibility
(Hardware requirements,
Integrations)
Choose the
• Validate your licenses
appropriate target
Cisco ISE Licensing Guide - Cisco
version
• Open bugs review :
• Use Bug Search Tool
(cisco.com))
• Engage with Cisco PS for
Software Analysis

BRKSEC-2889 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
Target Version : Do not forget

• Validate your upgrade path • Target the latest patch

• Avoid running ISE in production

2.7 3.2 without a patch

• Patch are cumulative

2.6 ? 3.2 • Allow a 2–3-week delay post-

patch release before production
Two-step upgrade : Perform the installation
biggest jump first Install Patch on ISE - Cisco

BRKSEC-2889 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
Choose your Upgrade Method

• Backup and Restore • GUI • CLI

• Recommended method

• Fast but more • Long but less • Longer and more

administration required administration required administration required
• Difficult to perform • Easy • Moderate difficulty

BRKSEC-2889 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
 Difference between Upgrade Method and
Upgrade Sequence
• Upgrade Sequence : In which order the ISE nodes are upgraded
• Common upgrade sequence steps: (except GUI Full Upgrade Method)

Upgrading the PMNT before the

SMNT avoids a persona change.

BRKSEC-2889 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
GUI Upgrade Methods

Split Upgrade Full Upgrade

• Step-by-step guide (added 2.6P10, 2.7P4, 3.0P3)
• Step-by-step guide with pre-checks
• Node upgrade sequence: One at a
time (Exception - up to 4 PSNs • Two steps upgrade :
simultaneously) • PPAN
• All the other nodes
- Basic knowledge about the ISE
installation and configuration
• Upgrade duration: Same for all
- Great for small / medium
deployments
deployment
• No Persona change
• Downtime: Required

BRKSEC-2889 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
New Split Upgrade Method New

• Coming with ISE 3.3 and 3.2 P3

• Nodes are now upgraded in Iterations
• Iterations can contain 15 nodes !
• Upgraded configuration is copied from
SPAN to other nodes
• Patch installation is available !
• Includes common Pre-Checks!

Understanding new split upgrade on Cisco ISE - Cisco

BRKSEC-2889 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
Pre-Checks List For Reference

• Repository Validation • Platform Support

• Bundle Download • Deployment Validation
• Memory check • DNS Resolvability
• Patch Bundle download • Trust Store Cert Validation
• PAN Failover Validation • System Cert Validation
• Scheduled Backup • Disk Space Check
• Configuration Data Upgrade • NTP Validation
• Services or Process Failures • Load Average Check

BRKSEC-2889 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
CLI Upgrade Method

• Better control over the upgrade

• Good the visibility on the
upgrade status
You can shorter the
• Requires a good knowledge of upgrade duration by
the upgrade process upgrading nodes in
Parallel !

Fallback method to a
failed upgrade GUI

BRKSEC-2889 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
Backup and Restore

• Not an upgrade, but node Only option available for Cloud

reinstallation deployments (AWS, Azure and OCI)
• Backup restoration on new
PPAN (old SPAN)
B&R can fully be automated
• Other Nodes configuration Upgrading ISE in the Cloud with
synchronized during cluster Automation – YouTube by
join Charlie Moreton
• Operation and deployment
knowledge level: Highest
required

BRKSEC-2889 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
Other upgrade methods ?

• Mix of CLI Upgrade and • Automated reconfiguration of

Backup & Restore ISE
• Upgrading the SPAN and PMNT • Install a fresh new blank config
nodes • Use APIs to recreate the
• Reinstall and join PSNs necessary configurations

BRKSEC-2889 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
How to estimate your upgrade duration ?

• Disclaimer: Estimated timings Operation Estimated duration

subject to environment Installing ISE 1- hour
specifics Restoring a configuration 30 minutes
backup (PPAN)
• Use the URT (Upgrade
Readiness Tool) to have a Synchronizing 30 minutes
configuration from PAN
better estimation !
Upgrading a PAN 2/3 hours

Improving the upgrade duration Upgrading an MNT 2/3 hours + 1 hour / 15

GB of operational data
can be achieved by cleaning
endpoints, users, and logs. Upgrading a PSN 1+ hour

BRKSEC-2889 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
Setting up a repository
• Required files for an upgrade (Backups, URT, Installation ISO,
upgrade bundle, patch). Make sure they are accessible and as
close as possible from the ISE nodes.
• Validating repositories through CLI on all nodes :

➢ show running-config | include repository

➢ show repository {repository_name}

• If the download of the Upgrade bundle download takes more

than 35 minutes, it might timeout

BRKSEC-2889 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
 Using the local disk as repository
You can use the Local Disk (disk) to store URT or Upgrade bundle.
Warning : the local disk space is limited !

• Configure the repository from CLI :

• Use the CLI the copy command to copy files to (config)# repository disk
(config-Repository)# url disk:
the local disk
• Use the dir command to list the files on the # copy ftp://{server}/{filename}
local disk or check the free space. disk:/

Since ISE 3.1, manage local

disk files from the GUI !
(Admin-System-Maintenance)

BRKSEC-2889 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
 Run the URT
Applies for all upgrade methods (except Full Upgrade and New Split Upgrade)

• Script: Run on SPAN

> application install URT repository
• Service impact: None […]
Checking ISE version compatibility
• Actions: Perform common - Successful
checks, check database Checking ISE persona
- Successful
compatibility with new version […]
Running data upgrade on cloned database
• Upgrade time estimation for each - Successful
node […]
Time estimate for upgrade
(Estimates are calculated based on size of
While the URT is running, do not config and mnt data only[…))
Estimated time for each node (in mins):
perform any persona change or ise30(STANDALONE):83
trigger backup.

BRKSEC-2889 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
What if the URT is unsuccessful ?

• The failure reason is clear

Fix the issue and rerun the URT to validate. Example :

• The failure is not clear :

Logs are stored on the localdisk and should be shared with TAC for analysis

BRKSEC-2889 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
 How to reduce upgrade duration (except B&R)
• You can purge MNT node operational
data using the ‘Purge data Now’ option
in the ISE GUI.
• Operational data logs are not
synchronized between the MNTs in
case of a persona change !

If the node was previously MNT and still hold Operational Data logs.
Use the following CLI command to purge logs on the node :
# application configure ise
[…] [3]Purge M&T Operational Data […]
# 3
[…] Enter days to be retained: 20

BRKSEC-2889 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
How to backup ?

• Backup options: Configuration (PPAN) or Operational (PMNT) from GUI or

CLI
• Restoration of configuration backup: Option to restore ADE-OS
• ADE-OS data: Hostname, IP address, NTP, running configuration, etc.

Issue: Slow or stuck backup

(Config or Operational) > application configure ise
[…]
Solution: Try canceling via GUI or [24]Force Backup Cancellation
use CLI command […]

BRKSEC-2889 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
Restoring your config backup in a lab !

• Consider setting up a lab VM of targeted ISE version

• Install latest patch
• Restore production environment backup (without ADE-OS)
• Successful restore indicates upgrade confidence
• Recommended: Planning some authentication tests

BRKSEC-2889 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
Pre-Upgrade validations (checklist) For Reference

• Backup • Clean
• Configuration, Operational Data, • Delete expired certificates
Network devices, endpoints (.csv) • Purge operational data, inactive
• Load balancers endpoints and guest accounts
• Export certificates and private keys
• Export internal CA certificates from CLI • Do not forget
• Perform Health Checks (since 2.6P8+)
• Take notes • Install latest patch (before the upgrade)
• AD Credentials and similar credentials • Disable PAN failover
• MDM credentials • Disable scheduled backups
• Profiler configuration for each PSN

BRKSEC-2889 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
Prepare the Maintenance Windows

• Use maintenance windows

• Yes, it is possible to upgrade in multiple maintenance windows

• Communicate about possible downtime

• Schedule extra-time ! (Estimate worst case scenario)

• Write a Method of Procedure document (MOP). Cisco CX

Professional Services can help writing such a document.

• Open a proactive TAC 48 hours before the operation !

BRKSEC-2889 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
Upgrading ISE is not
easy, unless you are
well prepared!

Cruising altitude
checklist | Jeffery
Wong
Pre-flight Checklist | David | Flickr
© 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public
Performing the
Upgrade
 • An ISE upgrade is long, but
do not rush it and take your
time to double check every
step.

• Any mistake :
To keep in mind
“This little maneuver’s
gonna cost us 51
years”
Cooper, Interstellar

BRKSEC-2889 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
Some Upgrade Best Practices

• Start your CLI upgrade using a Remote Console !

• Pre-upgrade file upload to ISE node using the following command :

# application upgrade prepare {Bundle Name} {Repository}
When ready start the upgrade using :
# application upgrade proceed

If an upgrade launched from the GUI takes longer than 4

hours, the upgrade might fail.
In that case, It’s recommended to upgrade via CLI.

BRKSEC-2889 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
Upgrading an ISE Virtual Machine

• If you need to reinstall ISE on multiple VMs in parallel, it is faster

to use the an ISO image than to use an OVA.

• Do not forget to Update Guest OS version

• Procedure (after the upgrade):
1. Shutdown
2. Change Guest OS
3. Start

• ISE disk size increase: Reinstallation only supported method

BRKSEC-2889 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
Upgrading an SNS Appliance

• CIMC and BIOS upgrades: Fix bugs, secure from vulnerability,

enhance hardware stability

• You can use the ISE Upgrade Maintenance window for CIMC
upgrade but it’s best to plan a dedicated one

• Backup and restore: ISO installation proximity to appliance/VM

necessary
• The faster method : use a bootable USB key (Since ISE 3.2, the
recommended software for bootable USB creation is Rufus).

BRKSEC-2889 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
How to monitor the upgrade process ?
Main ISE system logs will be included in ade/ADE.log log file.
To view the upgrade STEPs in live you can use :

> show logging system ade/ADE.log tail | include STEP

info:[application:install:upgrade:preinstall.sh] STEP 0: Running pre-checks
info:[application:operation:preinstall.sh] STEP 1: Stopping ISE application...
info:[application:operation:preinstall.sh] STEP 2: Verifying files in bundle...
info:[application:operation:isedbupgrade-newmodel.sh] STEP 3: Validating data before upgrade...
info:[application:operation:isedbupgrade-newmodel.sh] STEP 4: De-registering node from current deployment.
info:[application:operation:isedbupgrade-newmodel.sh] STEP 5: Taking backup of the configuration data...
info:[application:operation:isedbupgrade-newmodel.sh] STEP 6: Registering this node to primary of new deployment...
info:[application:operation:isedbupgrade-newmodel.sh] STEP 7: Downloading configuration data from primary of new
deployment...
info:[application:operation:isedbupgrade-newmodel.sh] STEP 8: Importing configuration data...
info:[application:operation:isedbupgrade-newmodel.sh] STEP 9: Running ISE configuration data upgrade for node
specific data...
info:[application:operation:isedbupgrade-newmodel.sh] STEP 10: Running ISE M&T database upgrade...
info:[application:install:upgrade:post-osupgrade.sh] POST ADEOS UPGRADE STEP 1: Upgrading Identity Services Engine
software...
info:[application:operation:post-osupgrade.sh] POST ADEOS UPGRADE STEP 2: Importing upgraded data to 64 bit
database...
BRKSEC-2889

BRKSEC-2889 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
After a node is upgraded

• Verify the ISE services are up and running :

• #show application status ise

Do not make any operation on the node before the having the
Application Server running !

• Validate the version installed

• #show version

BRKSEC-2889 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
What if the upgrade fails ?

The common logs to check for an error are ade/ADE.log and ise-
psc.log. You can view them using :
# show logging system ade/ADE.log
# show logging application ise-psc.log

• Collect support bundle through CLI using the following command :

# backup-logs {Backup Name} repository {Repository Name} public-
key
• Upload the file to your TAC case for a log analysis.

Cisco Worldwide Support Contacts - Cisco

BRKSEC-2889 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
Failures Remediation

SPAN Upgrade Failure : Non-PAN Upgrade Failure : PPAN Upgrade Failure :

• Failure before reboot : • Failure before reboot : • Failure before/after

reboot :
→ Node will automatically → Automatically joins back the
join back the old old deployment. → Fresh install and join the
deployment. Do not new deployment as SPAN
continue the upgrade Check with TAC or Fresh install
the node and join the NEW
deployment

• Failure after reboot : • Failure after reboot :

→ Fresh install the node → Fresh install the node and
and join back the old join the NEW deployment
deployment

BRKSEC-2889 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
One Common Mistake

• SPAN upgrade successful, but upgrade failing on other nodes

STEP 7: Importing configuration data...

% Error: Sanity test found some objects missing in CEPM schema...
% Warning: Sanity test found some indexes missing in CEPM schema. Please
recreate missing indexes after upgrade using app configure ise cli
% Error: Configuration database Schema Sanity failed!

Do not install any patch before finishing the upgrade on all nodes !
• Workarounds :
• Deregister the node before upgrading it, re-join the cluster after upgrade
• Use the Backup and Restore method

BRKSEC-2889 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
Warning on Field Notice FN 72499

• ISE 3.1: Supports RSA-PSS signature, incompatible with

Anyconnect on Windows
• Bug impact: Anyconnect versions 4.10.04065 and earlier can't
authenticate to ISE
• Fix: Upgrade to a fixed Anyconnect version
• Workaround: Disable RSA-PSS on PSNs using CLI
# application configure ise
[33]Enable/Disable/Current_status of RSA_PSS signature for EAP-TLS

BRKSEC-2889 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
Warning on URL Change for Smart Licensing

• From ISE 3.0 Patch 7, 3.1 Patch 5 and 3.2, the URL to check the
ISE licenses consumption has changed

• The new URL is : https://smartreceiver.cisco.com

• Work accordingly to authorize it in your proxy or firewall.

BRKSEC-2889 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
 For Reference

• Install latest Patch

• Install Patch on ISE – Cisco

• Check the Post-Upgrade

After Upgrade tasks :
Checklist • Step 5 of the Upgrade Journey
• Re-Join Active Directory
• Regenerate the Root CA chain
• Check Cipher suites
• Update Profiler Feed Service
• Restore Operational Data

BRKSEC-2889 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
Congratulations!
You've Successfully Upgraded Your ISE Deployment.

BRKSEC-2889 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
Conclusion
Conclusion

No magic trick to master an ISE Upgrade

• Important: Understanding ISE installation and upgrade process

• Key aspects: Proper planning, maintenance window usage,

downtime communication

• Assistance: Utilize Cisco CX services, proactive TAC engagement

BRKSEC-2889 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
Upgrading ISE is not
easy, unless you are
well prepared!

Source: NASA
© 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public
Thank you