MOP

SSR

CUBE IOS XE UPGRADE 16.12.8

Method of
Procedure

TABLE OF CONTENTS

Table of Contents.................................................................................................................... 1

OPTANIX MOP V1.0 1

Revision History...................................................................................................................... 1

Change details........................................................................................................................ 2

Affected devices...................................................................................................................... 2

Associated devices.................................................................................................................. 2

Contacts.................................................................................................................................. 2

Supplemental information....................................................................................................... 3

Terminal Servers.................................................................................................................. 3

Vendor Documentation........................................................................................................ 3

Purpose................................................................................................................................... 3

Description........................................................................................................................... 3

Reason for change............................................................................................................... 3

Technical impact/Risk............................................................................................................. 3

Requirements.......................................................................................................................... 3

Preliminary steps.................................................................................................................... 4

Pre-change output collection................................................................................................... 4

Implementation plan............................................................................................................... 4

Implementation.................................................................................................................... 4

Verification........................................................................................................................... 4

Backout Plan........................................................................................................................... 5

Backout Implementation...................................................................................................... 5

Backout Verification............................................................................................................. 5

post-change output collection................................................................................................. 5

Final Steps.............................................................................................................................. 5

REVISION HISTORY
Revisi Tea
Date Engineer Notes
on m

OPTANIX MOP V1.0 2

CHANGE DETAILS
Information Details
Optanix Ticket #
Customer Ticket #
Vendor Ticket #
Estimated Duration of
Pre-Checks
Estimated Change
Duration

AFFECTED DEVICES

The following list contains devices that will be modified:

IP Device
Device Name
Address Type/Function

ASSOCIATED DEVICES

The following list contains devices that will NOT be modified but may be indirectly impacted.

Device Name IP Address Device Type/Function

CONTACTS

The following list contains all individuals involved with the change (including vendors):

Name Organizati Role Email Address Phone Number Primar

on y

OPTANIX MOP V1.0 3

 Contac
t

SUPPLEMENTAL INFORMATION

TERMINAL SERVERS

VENDOR DOCUMENTATION

 https://software.cisco.com/download/home/284358776/type/282046477/release/Gibraltar-
16.12.8

PURPOSE

DESCRIPTION

 IOS will be upgraded on ALL CUBEs

REASON FOR CHANGE

 Investigation done in Case 8833 found that recording legs of IVR calls were experiencing
an error and the recording forked media was not properly disconnected. The recordings
are being created however some files are much larger than others and this is due to a
BUG from Cisco.
 https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvq57205
 Investigation from Case 9043 found that the 16.9.8 IOS created a memory leak
 https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvt12913
 IOS 16.9.5 kept the memory leak running
 SSR 11422 was completed to add smart licensing to the CUBEs prior to upgrading to an
IOS version that requires it.

TECHNICAL IMPACT/RISK

Outline the expected technical impact and/or risk associated with the implementation of this
change. Example:

REQUIREMENTS
Function Required Details (if function required)
Onsite support ☒ In the event an issue happens and we need

OPTANIX MOP V1.0 4

 console access to the CUBEs
Vendor or TAC ☐

Hardware RMA ☐

Customer/User/App ☒ Customer will need to be present to ensure no

Testing issues arise during failover.
WebEx or ☒
Conference Bridge
Customer contact ☒
required before
change
Additional ☐
Requirement

PRELIMINARY STEPS
 Review all steps in the implementation plan before proceeding
 If applicable, create outage for associated device(s) to suppress alarms
 If required, join bridge
 Update case with details from the above steps taken and change case status to “Work in
progress”.
 Notify all listed in the Contact(s) section via email, please follow the format specified in
the Change Management documentation.
 Required Files

Details
Description : Cisco ISR 4400 Series IOS XE Universal
Release : Gibraltar-16.12.8
Release Date : 20-Sep-2022
FileName : isr4400-universalk9.16.12.08.SPA.bin
Min Memory : DRAM 4096 Flash 8192
Size : 613.13 MB ( 642909463 bytes)
MD5 Checksum : 936fb636ed845d8120afd763a77db3fc
SHA512 Checksum :

eb13266545b13b9bcb4a69ffdefa759e8bdc13f88feddbaf3ebd11ce6235ba21c73fccd58a7cc3
87711212ecd801cd89e59a69522c56106864791f52ce81ecf7
isr4400-universalk9.16.12.08.SPA.bin
100% 613MB 162.7KB/s 1:04:19
sampson@A1PPOPT01:~/SSR12111$ scp isr4400-universalk9.16.12.08.SPA.bin
aric.nellis@10.10.70.41:isr4400-universalk9.16.12.08.SPA.bin
Password:
isr4400-universalk9.16.12.08.SPA.bin
100% 613MB 1.2MB/s 08:23
Connection to 10.10.70.41 closed by remote host.
sampson@A1PPOPT01:~/SSR12111$ scp isr4400-universalk9.16.12.08.SPA.bin
aric.nellis@10.20.70.41:isr4400-universalk9.16.12.08.SPA.bin
Password:
isr4400-universalk9.16.12.08.SPA.bin
100% 613MB 156.1KB/s 1:07:02

OPTANIX MOP V1.0 5

Connection to 10.20.70.41 closed by remote host.

PRE-CHANGE OUTPUT COLLECTION

 Take the necessary backups/screenshots of the current configurations that will be
modified and any informative “show” commands. Please reference, MOP Workflow for
more details.
 Filename should utilize a similar format to the below:
[devicename]_[pre,post]-change_[date].log
Date format: yyyy.mm.dd
Example: Router001_pre-change_2014.12.3.log

Cisco Voice Gateway/CUBE - Pre-Change Output Collection:

The devices have been listed under “Affected Devices” on previous steps. The actual commands
and details are found in: https://tacconnect > Home > Homepage > Health Checks
Perform a Pre-change Health Check from each device as follows: (Gather, Review and
Document)
System running configurations
Software Versions
Hardware Inventory and Status
Licensing Status
Interface/Controller Status
Local Flash/Storage details
All voice interface Status (FXO/PRI/FXS)
System Resources health (CPU/Memory)
Signaling Protocol Status (MGCP/SCCP/H323/SIP)
Active Calls Report (if any)
CUBE HA Status (if enabled)
Commit the configurations in flash to the startup configurations.
Client/Server Health Checks:
Gather/Review/Document the following from the “server/CUCM”:
· Gateway/End-Point Registration counts.
o Registration details for:
§ Gateway (MGCP/H323/SCCP)
§ SIP Trunks
§ Media Resources (SCCP)
IOS Backups:
· The data gathered and documented above includes a copy of the configurations, which will be
used as a backup.

XMF Health Checks:

Show wsapi registration all
Show call media-forking

OPTANIX MOP V1.0 6

IMPLEMENTATION PLAN

IMPLEMENTATION
DENVER CUBES
1. Save a copy of run-configuration to flash prior to any change
2. Read backout plan if downgrade is needed
3. DO NOT roll over to 03.13.x image which will wipe ssh key from the device and block ssh
access
4. The changes will be done on the secondary cubes first as they are not processing calls

5. Make sure there is no active call on the device

Show call active voice summary
6. If calls are present stop sip call processing

config t
voice service voip
sip
call service stop

7. Wait for any active calls to stop.

8. Next we will modify the boot variable for the IOS

!
conf t
!
no boot system flash bootflash:isr4400-universalk9.16.09.08.SPA.bin
no boot system flash bootflash:isr4400-universalk9.16.09.05.SPA.bin
no boot system flash bootflash:isr4400-universalk9.16.09.02.SPA.bin
!
boot system flash bootflash:isr4400-universalk9.16.12.08.SPA.bin
boot system flash bootflash:isr4400-universalk9.16.09.05.SPA.bin
!
wr mem
!

9. Verify the configuration change

!
show run | inc boot system
!
show bootvar
!

10. Save and reload the cube

OPTANIX MOP V1.0 7

 !
wr mem
reload reason
!

11. Test connectivity to the device until it returns

a. Make incoming/outgoing calls, if possible
12. Check XMF registration status
a. Show wsapi registration all
b. It will take a while for the xmf to register to the application servers
13. Make incoming/outgoing calls, if possible
a. Confirm DTMF works for incoming calls.
Show call media-forking
b. During previous change, it was found that XMF service has to be restarted on the
cubes.
Conf t
Uc wsapi
Provider xmf
Shut
No shut
c. Confirm XMF registration status again
d. In case the inbound DTMF still has the issue after XMF was restarted on the cubes,
engage application team to check the XMF application server
14. Repeat these steps on the other secondary cube once the first is completed
successfully.
15. Make sure site is ready prior to route calls from to
16. Once both Cubes(standby) are completed then we will begin on (active cubes)
17. These are the active cubes we will take steps to fail calls over to the Denver side.

!
conf t
voice service voip
sip
call service stop
!
end
!
wr mem
!

18. Apply this to both Waukee cubes

19. Monitor calls on and using the following.

!
Show call active voice sum
!

20. No calls should be present on the Cubes and all call processing should be on the
Cubes before proceeding
a. Test with the customer if there are no live calls to verify.

OPTANIX MOP V1.0 8

21. Repeat steps 5 – 14 on the Cubes to upgrade them.
22. Test connectivity to the device until it returns
23. Once the upgrades are completed return call processing to normal

!
conf t
voice service voip
sip
no call service stop
!
end
!
wr mem
!

24. Verify the min-se timer on all cube devices

a. Show run | inc min-se
b. The expected return is min-se 28800
c. To modify if this is not the returned value move to step 25
25. Modify the min-se timer back to co-op value

!
conf t
voice service voip
sip
min-se 28800
!
end
wr mem
!

26. Verify that calls return to the cubes and that there are no issues.
27. If at any point during implementation there are unexpected results that may cause
impact to production:
a. Spot check the appliance to assist in determining the impact
b. Take steps noted in the Unsuccessful Implementation section under Final Steps
c. Call the customer listed in the Contact(s) section
d. Do your best to remediate the issue

VERIFICATION
1. Show platform
2. Show call active voice sum or brief
3. Show version
4. Show wsapi registration all
5. Show call media-forking
6. Confirm DTMF works fine for incoming calls
7. If verification fails:
a. Do your best to remediate the issue
b. If unable to remediate, jump to the “Back-out Plan” section
8. If verification succeeds, jump to the “Post Output Collection” section

OPTANIX MOP V1.0 9

BACKOUT PLAN

BACKOUT IMPLEMENTATION
1. The customer will need to approve and backout to be undergone.
2. There is no backout plan other than if issues arise contact Cisco Under SR 692867129 re-
open and raise severity

POST-CHANGE OUTPUT COLLECTION

 Take the necessary backups/screenshots of the current configurations and any
informative “show” commands that were run in the “Pre-change Output Collection”
section. Verify for any differences between the pre and post output.
 Filename should utilize a similar format to pre-change filenames

FINAL STEPS
 Ensure that all debugs and elevated traces are disabled unless explicitly stated
otherwise
 Please refer to the Operations documentation, Change Management, and complete all
steps to finalize the change and to communicate effectively on the change
status/completion.

OPTANIX MOP V1.0 10