Node Upgrade Guide

Isilon
X400

Upgrade a node

September 2016

l Upgrading nodes........................................................................................................2
l Preparing for the upgrade...........................................................................................3
l Performing the upgrade............................................................................................11
l Reviewing the upgrade.............................................................................................26
l Roll back a failed upgrade........................................................................................28
l Where to go for support............................................................................................29
Node Upgrade Guide

Upgrading nodes
You can upgrade the drives, memory, and NICs in most EMC Isilon nodes.
The process to upgrade nodes in a cluster consists of three stages.
Preparing for the upgrade
An upgrade procedure requires significant planning and preparation. Steps in the
process, such as smartfailing drives from multiple nodes, can take several weeks to
complete. Before the actual maintenance window, you must also obtain a hardware
upgrade package for each node you plan to upgrade. It is important that you understand
this procedure and communicate its full impact to everyone who might be affected.
Performing the upgrade
After you prepare the cluster for work, you must upgrade the nodes during the agreed
upon maintenance window. A maintenance window is necessary, because full cluster
availability during the upgrade procedure is not guaranteed. You must adhere to all EMC
Isilon safety and quality guidelines as you perform each step of the procedures for
removing and installing node components.
Reviewing the upgrade
After nodes are upgraded, you must review the status of the cluster and confirm that all
configuration changes were successful and that all new storage pools are established
and healthy.

Downgrading a node
You can downgrade configure-to-order (CTO) components in a node.
The procedure to perform a downgrade is exactly the same as the procedure to perform
an upgrade. Follow the steps to prepare, perform, and review all changes to the node
configuration.

Working with clusters in SmartLock compliance mode

Clusters running in SmartLock compliance mode require a sudo prefix to run root
commands.
If a cluster is running in SmartLock compliance mode, root access is disabled on the
cluster. Because of this, you can run some commands only through the sudo program.
Prefixing a command with sudo enables you to run commands that require root access.
For example, if you do not have root access, the following command fails:

isi drivefirmware status

However, if you are on the sudoers list, the following command succeeds:

sudo isi drivefirmware status

Compliance mode commands that require changes beyond the sudo prefix are noted in
the procedure steps.
For more information on the sudo program and compliance mode commands, see the
OneFS CLI Administration Guide.

2 X400 Node Upgrade Guide

 Upgrade a node

Preparing for the upgrade

Before you upgrade nodes in a cluster, open a support case, prepare the environment,
and obtain a hardware upgrade package.

Schedule an upgrade maintenance window

When you perform an upgrade, the procedure will interrupt the cluster workflow and
performance might be affected. Schedule a maintenance window to avoid unexpected
disruptions.
The cluster will remain online during the upgrade procedure, but full cluster availability
cannot be guaranteed. Nodes will be rebooted one at a time during the process, and will
also be removed and added to storage pools.
Procedure
1. Schedule a maintenance window based on the number of nodes that you will
upgrade.
The window should allow for up to one hour per node.

Request a hardware upgrade package

You must request hardware upgrade kits, and a hardware upgrade file, from the Isilon
Verification team.
For hardware upgrades, in addition to the physical hardware upgrade kit for each node,
you will also need a customized upgrade package file with a specific serial number .
Procedure
1. Do one of the following:
l Run the following command to view a current diagnostic log for the cluster:
isi_gather_info
l Run the following command to obtain upgrade receipts from the upgraded nodes:
isi_inventory_tool --display --itemType=receipt
Record the system output.
2. Gather the following cluster information:
l Sales order number
l Cluster name
l Node numbers
l Node serial numbers
l OneFS version

3. Request a download link for the hardware upgrade package from the Isilon
Verification Team by sending the information you gathered in previous steps to
cto.upgrades.validation@emc.com.
Requests for download links can take approximately 7 business days. These requests
must be made before you begin the upgrade process.

Preparing for the upgrade 3

Node Upgrade Guide

Note

EMC Isilon Technical Support cannot help with the creation or correction of hardware
upgrade packages, or changes to receipts. For these requests, contact the Isilon
Verification Team.

Download the hardware upgrade package

Before you upgrade your hardware, obtain a hardware upgrade package from your Isilon
Account Team. The hardware update package updates the configure-to-order (CTO) and
as-built information on the node, then forwards the updated information to Isilon
Technical Support.
Procedure
1. Obtain a hardware upgrade package from your Isilon Account Team.
2. Note the name of the hardware upgrade package. You will use the name for other
commands.
Package names follow this convention:
IsiHW_Package_ [order number] _ [node serial number] _ [date and time
stamp] .tgz
For example,
IsiHw_Package_ORD108024_SX2003011184736_201507072125.tgz

3. Place the hardware upgrade package on the cluster through a network drop, or by
asking someone at the cluster site to place the package for you. If neither of these
options is available to you, contact Isilon Technical Support for assistance.

Install the hardware upgrade package on the node

You must place an upgrade package on each node that is receiving an upgrade, unpack
the package, and then install it.
Procedure
1. Take the upgrade package that was provided by your Isilon Account Team and place it
on the node that the package was created for.
Use a network drop or have the customer place the package for you.

2. Unpack the update by typing the following command:

tar xzvf IsiHW_Package_<order-number>_<node-serial-number>_<date-
time-stamp>.tgz

3. Change directories to where the package was placed.

4. Install the update package. Depending on your version of OneFS, type one of the
following commands:
OneFS 8.0 or later
isi upgrade patches install<order-number>_<node-serial-number>_<date-
time-stamp>.tar

Earlier than OneFS 8.0

isi pkg install --forced_local IsiHW_Package_<order-
number>_<node-serial-number>_<date-time-stamp>.tar

4 X400 Node Upgrade Guide

 Upgrade a node

As the package installs, the following messages appear:

Preparing to install the package...

Checking the package for installation...
Installing the package
Committing the installation...
Package is committed.

If the node does not perform as expected, contact Isilon Technical Support.

5. Repeat these steps for each node that you are upgrading.

Note

Your Isilon Account Team will generate an individual upgrade package for each node
you are upgrading. Make sure to install the correct upgrade package on each node.

Review the upgrade receipt

Verify that the receipt file in the hardware upgrade package is consistent with the defined
upgrade path and the hardware order.
Procedure
1. Change directory to the location of the upgrade package by running the following
command:

cd /var/crash/cto/hwPackages/IsiHW_Package_<order-number>_
<node-serial-number>_<date-time-stamp>

2. Display the receipt file by running the following command:

./isi_cto_upgrade --overview

Note

If the cluster is in SmartLock compliance mode, display the receipt file by running the
following command:

sudo /usr/bin/isi_HWtools/isi_cto_upgrade --overview

Note

You can execute this command in advance of the upgrade engagement. This
command will not make changes to the node.

The command output will confirm verification of the package integrity, node configure-
to-order (CTO) capability, chassis VPD, and the node serial number. The output will
also display the receipt that was generated before the upgrade next to the receipt that
was generated after the upgrade.

Note

You might see some noncritical error messages as part of the output. If you are not
sure whether it is safe to continue, contact EMC Isilon Technical Support.
3. Compare the command output of the receipt that was generated before the upgrade to
the command output of the receipt that was generated after the upgrade.

Request a hardware upgrade package 5

Node Upgrade Guide

The content of the receipt will be similar to the following example:

<receipt
receiptFileName="30301230_SX4001013160081.xml"
receiptFormatVersion="1.2" date="2014-01-23T01:40:03Z">
<bom>
<sbb qty="1" desc="SBB, BASE, DDR, X400" part="610-0005-03"/>
<sbb qty="1" desc="SBB, 3.5" 66TB HDD / 1.2TB SSD EMLC"
part="611-0075-02"/>
<sbb qty="1" desc="SBB, Memory, 48GB DDR3, 12x4G"
part="612-0006-01"/>
<sbb qty="1" desc="SBB, 10GbE, Dual-port, SFP+, without Optics"
part="613-0002-01"/>
</bom>
</receipt>

Results
If the receipt from before the upgrade is not consistent with the receipt from after the
upgrade, or if neither are consistent with the defined upgrade path and the hardware
order, contact your Account Team and request a corrected upgrade package.
If you have issues that are not related to the receipt, contact EMC Isilon Technical
Support.

Review storage pools configuration

As you upgrade nodes in a cluster, OneFS will organize the upgraded nodes into a new
storage pool.
After you upgrade a node, OneFS recognizes the new node configuration and creates a
new node pool for the upgraded node. As long as there are fewer than three nodes in the
new node pool, OneFS will only read data from that node, not write to it. After you
upgrade a third node and there are three nodes in the new node pool, OneFS will write
data to those nodes.
You might experience degraded performance as nodes are reprovisioned to new node
pools, but data on the cluster will remain available.
Before you begin an upgrade engagement, review the cluster's storage pool configuration
to determine which policies you will need to change after the upgrade is complete.
For more information about storage pools, see the OneFS CLI Administration Guide.

Adding new nodes during an upgrade

If you are adding new nodes to a cluster as part of a hardware upgrade engagement,
review the configuration of the nodes to determine when you should add them to the
cluster.
If the nodes that you are adding to the cluster have the same hardware configuration as
the existing nodes prior to the upgrade, add the new nodes to the cluster before you
upgrade.
If the nodes that you are adding to the cluster have the hardware configuration that the
existing nodes will have following the upgrade, add the new nodes to the cluster only
after you upgrade the existing nodes and confirm that the cluster is healthy.
Following these guidelines will prevent OneFS from creating new node pools before the
hardware upgrade is completed, which will reduce the complexity of the environment.

6 X400 Node Upgrade Guide

 Upgrade a node

Review scheduled jobs

Review scheduled jobs that might cause performance issues during the upgrade.
Pause any running jobs to minimize cluster performance issues during the upgrade
process.
If you are upgrading drives, pause all jobs before you smartfail the drives, to enable the
smartfail process to complete as quickly as possible.
You can restart the jobs after the upgrade is complete.

Confirm SSD compatibility

If you are installing a new solid state drive (SSD) into a cluster, you must confirm that the
drive you have onsite is compatible with the cluster.
See the Isilon Supportability and Compatibility Guide to determine whether the drive you
have is compatible with the cluster.

Gather logs
Before you begin any maintenance on a cluster, gather cluster logs.
You must collect cluster logs before all maintenance procedures. Cluster logs provide
snapshots of the cluster, which you can review to make sure that maintenance is
successful.
Procedure
1. Open a secure shell (SSH) connection to any node in the cluster and log in.
2. Gather cluster logs by running the following command:
isi_gather_info

Review the current hardware status

View the status and configuration information for memory, drives, and NICs that are
currently installed.
Capture this pre-upgrade information to review after the upgrade is complete.
Procedure
1. Change directory to the location of the upgrade package by running the following
command:

cd /var/crash/cto/hwPackages/IsiHW_Package_<order-number>_
<node-serial-number>_<date-time-stamp>

2. View the current hardware configuration by running the following command:

./isi_cto_upgrade --overview

Review scheduled jobs 7

Node Upgrade Guide

Note

If the cluster is in SmartLock compliance mode, view the current hardware

configuration by running the following command:

sudo /usr/bin/isi_HWtools/isi_cto_upgrade --overview

3. Confirm that the current amount of installed RAM matches the hardware upgrade
receipt, and that all DIMMs are healthy.
4. Confirm that the installed drives match the hardware upgrade receipt, and that there
are no failed drives.
5. Confirm that the installed network interfaces match the hardware upgrade receipt,
and that all NICs are healthy.

Open a support case

Open a support case for the customer prior to the upgrade engagement.
Procedure
1. Open a new support case with EMC Isilon Technical Support.
An open support case will save time if an issue arises during the upgrade procedure.
2. Attach your engagement plan and any other supporting documentation to the support
case.
Your engagement plan should include expected changes to storage pools and
rollback options, if you encounter an issue.

Smartfail drives to be upgraded

If you are upgrading drives as part of an upgrade engagement, smartfail the drives in
advance.
Before you begin

Note

You should not arrive for a drive upgrade engagement until all the drives that you are
going to upgrade, in all nodes, are smartfailed.

Before you smartfail multiple drives, check the amount of free space available on the
cluster. Follow best practices related to available space and protection levels on the
cluster.
Refer to the following documentation for more information:
l Best practice guide for maintaining enough free space on Isilon clusters and pools
l How to determine if an isilon cluster is in a window of risk for data loss
If you are upgrading to SSDs, smartfail drives in the lowest drive bay numbers. For
example, if you are adding two SSDs to each node, smartfail the drives in bays one and
two.

8 X400 Node Upgrade Guide

 Upgrade a node

Figure 1 Front drive bays

1 2 3 4

5 6 7 8

9 10 11 12

13 14 15 16

17 18 19 20

21 22 23 24
GHE-000066

Figure 2 Back drive bays

25 26 27 28

29 30 31 32

33 34 35 36
GHE-000067

Smartfail all the drives that will be upgraded, one node at a time. Wait for the smartfail
process to complete on one node, then proceed to the next node.
Procedure
1. Open a secure shell (SSH) connection to the node that contains the drive you want to
remove. Type the command:
ssh <cluster-name>-<node-number>

2. Confirm the node number and bay number of the drive you want to remove.
Depending on your version of OneFS, type one of the following commands:
OneFS 8.0 or later
isi devices drive list

Earlier than OneFS 8.0

isi devices

Note

Note the node number and bay number of the drive you want to remove. If the drive
you need to remove has already been smartfailed automatically, you can skip the next
step. A smartfailed drive will display a status of REPLACE or, if the node was restarted
after the smartfail, a status of EMPTY. A smartfailed SED drive might display a status
of ERASE. This status also indicates the node is ready to be replaced.

3. Using the node number and bay number of the drive, smartfail the drive by typing one
of the following commands:
OneFS 8.0 or later
isi devices drive smartfail <bay-number> --node-lnn <node-number>

Earlier than OneFS 8.0

isi devices -a smartfail -d <node-number>:bay<bay-number>

If the drive does not respond to the smartfail command, do not remove it from the
node. Contact Isilon Technical Support.

Smartfail drives to be upgraded 9

Node Upgrade Guide

4. Confirm that the drive was smartfailed by repeating step two.

CAUTION

Do not replace the drive until you have confirmed that the smartfail process is
complete.

Results
Repeat this procedure for every drive, in every node that you are upgrading.
The length of time that it takes for the smartfail process to complete depends on the size
of the cluster, drive capacity, and the cluster workflow.

Install a drive support package

For clusters running OneFS 7.1.1 and later, install the latest drive support package to
update your drive firmware to the latest supported revision.
Before you begin
See the Considerations for installing the latest drive support package section before you
begin the installation.
Procedure
1. Go to the EMC Support page that lists all the available versions of the drive support
package.
2. Click the latest version of the drive support package and download the file.

Note

See the Considerations for installing the latest drive support package section in order
to select the appropriate variant of the package.

3. Open a secure shell (SSH) connection to any node in the cluster and log in.
4. Create or check for the availability of the directory structure /ifs/data/
Isilon_Support/dsp.
5. Copy the downloaded file to the dsp directory through SCP, FTP, SMB, NFS, or any
other supported data-access protocols.
6. Unpack the file by running the tar command.
For example, based on the variant selected for the drive support package, unpack the
package by running one of the following commands:

tar -zxvf Drive_Support_<version>.tgz

tar –zxvf Drive_Support_<version>_No_SSD.tgz
7. Install the package by running the isi_dsp_install command.
For example, based on the variant selected for the drive support package, install the
package by running one of the following commands:

isi_dsp_install Drive_Support_<version>.tar
isi_dsp_install Drive_Support_<version>_No_SSD.tar

10 X400 Node Upgrade Guide

 Upgrade a node

Note

l You must run the isi_dsp_install command to install the drive support
package. Do not use the isi pkg command.
l The installation process takes care of installing all the necessary files from the
drive support package followed by the uninstallation of the package. You do not
need to delete the package after its installation or prior to installing a later version.

Reboot the cluster

With explicit permission from the customer, you can reboot the cluster to make sure the
status of the cluster is healthy.
Rebooting the cluster can flush out unexpected hardware alerts.

Note

Although rebooting the cluster can help avoid certain issues, taking this step might not
be possible in your environment. And, although this step is recommended, it is optional.

Procedure
1. Reboot the cluster according to EMC Isilon best practices for restarting entire clusters.

Performing the upgrade

Upgrade nodes with one or more new components.

CAUTION

Perform the upgrade procedure on only one node at a time. Working on multiple nodes
in parallel might lower the protection level of the cluster, put data at risk, and lead to the
interruption of client workflows.

Run the pre-upgrade script

After the hardware upgrade package is installed on the node, you must run the pre-
upgrade script.
Before you begin
If the hardware upgrade package is not installed, follow the procedure for installing a
hardware package on a node before you begin this procedure.
Procedure
1. Change to the correct directory by running the following command:

cd /var/crash/cto/hwPackages/IsiHW_Package_<order-number>_
<node-serial-number>_<date-time-stamp>

The directory name includes the package info.

2. To run the pre-upgrade script, run the following command:

./isi_cto_upgrade --pre

Reboot the cluster 11

Node Upgrade Guide

Note

If the cluster is in SmartLock compliance mode, run the pre-upgrade script by running
the following command:

sudo /usr/bin/isi_HWtools/isi_cto_upgrade --pre

Messages appear as the script confirms the following information:

a. CTO capability
b. Upgrade order number
c. Node serial number
d. Current node hardware configuration
The command output is similar to the following example:

Verifying package integrity

Package integrity is good
Beginning pre-installation actions
Node CTO capability verified
Upgrade is for Order 30548476, Node S/N SN4002313370143
Chassis VPD verified
Node S/N verified
Node CTO configuration verified
Importing SoftAVL (./Soft-AVL_201412020001.xml)
Success
Importing local /etc/gconfig/probe_config.0.gc (/etc/gconfig/
probe_config.0.gc)
Success
Importing local /etc/gconfig/probe_config.1.gc (/etc/gconfig/
probe_config.1.gc)
Success
Importing local /etc/gconfig/probe_config.gc (/etc/gconfig/
probe_config.gc)
Success
...

Note
The upgrade script displays a list of missing and found components. The missing
component list contains the new parts that you will add to the node. The found
component list contains the parts that you will remove. You must accept this message
to continue. You might see non-critical error messages as part of the output. If you do
not feel that it is safe to continue, contact EMC Isilon Technical Support.

3. Once the pre-upgrade script concludes, you are prompted to power down the node:

Ready to power down the node? (y/n)

4. Type y to power down the node.

Upgrading a drive
If you are upgrading drives as part of an upgrade engagement, remove the smartfailed
drives.
Remove the old drives only. Do not install the new drives until you have powered the
node back up.

12 X400 Node Upgrade Guide

 Upgrade a node

Access the drives

You can access drives by removing the panels that cover the drive bays.
Procedure
1. To access drives at the front of the node, remove the front panel.
There is a release button on each side of the front panel. To remove the panel, press
in the panel release buttons while pulling the panel away from the node.

1. Front panel

2. To access drives at the back of the node, remove the rear EMI shield.
There is a vertical handle at each end of the shield. To remove the shield, press the
handles toward the center of the shield while pulling away from the node.

1. Rear EMI shield

Upgrading a drive 13
Node Upgrade Guide

Remove the drive to be upgraded

Drives are secured in the node. You must release the locking handle before removing a
drive.
Before you begin
Only remove the drives that you smartfailed in preparation for the upgrade. If you are
upgrading to SSDs, the smartfailed drives should be in the lowest drive bay numbers. For
example, if you are adding two SSDs to each node, the smartfailed drives should be in
bays one and two.
Figure 3 Front drive bays

1 2 3 4

5 6 7 8

9 10 11 12

13 14 15 16

17 18 19 20

21 22 23 24
GHE-000066

Figure 4 Back drive bays

25 26 27 28

29 30 31 32

33 34 35 36
GHE-000067

Procedure
1. To remove a drive, pull the locking handle on the drive toward you.
The drive releases from the node.

14 X400 Node Upgrade Guide

 Upgrade a node

1. Locking handle 2. Drive bay

Upgrading a NIC or DIMMs

If you are upgrading a NIC or DIMMs as part of an upgrade engagement, you will need to
access the inside of the node.
If you are only upgrading drives, skip ahead to powering up the node. Do not disconnect
and open the node.

Slide the node out of the rack

Slide the node away from the rack to access the contents of the node.
Procedure
1. Label the InfiniBand, ethernet, and power cables connected to the back of the node to
ensure that they are reconnected correctly.
2. Disconnect all cables from the back of the node.

Note

If there are transceivers connected to the end of your IB or ethernet cables, make sure
to remove them with the cables. If you are using fiber ethernet cables, you will need to
disconnect the cable from the transceiver, then remove the transceiver from the node.

3. Remove the node front panel.

4. Remove the retaining screws that secure the node to the rack cabinet.
5. Slide the node from the rack cabinet to fully extend the slide rails and provide clear
access to the node. Do not remove the node from the slide rails.

Upgrading a NIC or DIMMs 15

Node Upgrade Guide

DANGER

Slide the node out from the rack slowly. Do not extend the rails completely until you
confirm that the node is latched and safely secured to the rails.

Remove the node top panel

You remove the top panel to gain access to the contents of the node.

WARNING

Properly ground yourself to prevent electrostatic discharge from damaging the node. For
example, attach an ESD strap to your wrist and the node chassis.

Procedure
1. Loosen the captive screw that secures the node top panel.
2. Slide the top panel toward the rear of the node, and then lift the top panel to access
the node interior.

Remove the cross bracket

Removing the cross bracket provides clear access to the inside of the node.
Procedure
1. Locate the cross bracket within the node.

16 X400 Node Upgrade Guide

 Upgrade a node

Figure 5 Cross bracket

2. Remove the cross bracket by pressing on the side of the node chassis where the cross
bracket is connected. Unhook the cross bracket from the chassis, then lift straight up
to unhook the other side of the bracket.

Remove the air baffle

In order to gain full access to the internal components of the node, you must remove the
air baffle.
Procedure
1. Locate the air baffle within the node.

Upgrading a NIC or DIMMs 17

Node Upgrade Guide

Figure 6 Air baffle

2. Raise the front end of the air baffle, unhook the tabs at the back end of the baffle, and
then lift the baffle out of the node.

Upgrading DIMMs
If you are upgrading DIMMs as part of an upgrade engagement, remove the old DIMMs
and replace them with the new DIMMs.
If you are installing a different amount of DIMMs than you are removing, make sure that
the new DIMMs are installed in the same color slots and in the same bank from which the
old DIMMs were removed.

Remove the DIMM to be upgraded

You must remove the DIMM that you are upgrading from the node.
Procedure
1. Press down on the DIMM locking arm on each side of the DIMM to release the DIMM
from the slot.

CAUTION

If you are replacing a DIMM in slot P2 DIMM 3A or P2 DIMM 3B, remove the network
interface card (NIC) to allow enough space to remove the DIMM without damaging
the NIC.

18 X400 Node Upgrade Guide

 Upgrade a node

1. DIMM

Install the new DIMM

You must install the new DIMM in the slot from which you removed the old DIMM.
Procedure
1. Remove the new DIMM from the antistatic package.
2. Locate the open slot that the old DIMM was removed from. Align the notch in the
DIMM with the tab of the open slot and press down firmly on both ends of the DIMM
until the two arms lock into place, securing the DIMM.

Note

Install the new DIMM in the empty slot that used to hold the old DIMM. A DIMM that is
installed in another open slot runs the risk of not being recognized by the system.

Upgrading a NIC
If you are upgrading the NIC as part of an upgrade engagement, remove the NIC and
replace it with the new card.

Upgrading a NIC or DIMMs 19

Node Upgrade Guide

Remove the NIC to be upgraded

You must remove the NIC that you are upgrading from the node.
Procedure
1. If necessary, disconnect any cables from the NIC.
2. Remove the mounting screw that secures the NIC to the node.
3. Remove the NIC from the node.

1. Mounting screw 2. NIC

Install the new NIC

After the original NIC is removed, you can install the new NIC.
Procedure
1. Insert the new NIC in the empty card slot from which you removed the original NIC.
Check to ensure that the card is fully seated in the motherboard connector.

20 X400 Node Upgrade Guide

 Upgrade a node

2. Secure the card to the back panel of the node with the mounting screw.
3. Reconnect any cables that you removed from the card.

Install new transceivers and cables

Install new front-end cables and transceivers if necessary for the new NIC.
Procedure
1. Locate front-end cables at the back of the node.
2. If the upgraded NIC requires new cables, replace the front-end cables with the new
cables.
3. Remove all dust covers and plugs from the new transceiver.
4. Insert the cable connector into the transceiver until it clicks.
Transceivers are keyed to ensure proper connection to the cable. If the transceiver
does not connect easily to the cable, check to be sure that you are aligning the
transceiver correctly.
5. Repeat these steps to attach the new transceiver to other front end cables as
necessary.

Install the air baffle

You must replace the air baffle by inserting the baffle tabs into slots in the chassis.
Procedure
1. Hook the tabs on the back end of the air baffle into the metal slots at the back of the
node.
2. Lower the front of the air baffle back into its original position within the node.

Install the cross bracket

You must install the cross bracket by hooking it to the bracket holes on the interior of the
node and then snapping the other end into the chassis wall.

WARNING

The cross bracket sits directly above the boot drives. Use caution when installing the
cross bracket so that the boot drives are not dislodged or damaged.

Install the node top panel

You must secure the top panel onto the node.
Procedure
1. Place the top panel on the node so that the front edge of the top panel is about one
inch behind the drive bays, and then slide the top panel forward into place.

WARNING

The chassis intrusion switch can be damaged if the top panel is slid too far back on
the node.

2. Tighten the captive top panel screw to secure the top panel to the node.

Upgrading a NIC or DIMMs 21

Node Upgrade Guide

Return the node to the rack

Return the node to the rack after all work is complete.
Procedure
1. Slide the node back into the rack cabinet.

WARNING

Slide the node slowly so you do not slam the node into the rack and damage the
node.

2. Reconnect the ethernet, InfiniBand, and power cables to the back of the node.
3. Secure the node to the rack cabinet.
4. Replace the node front panel.

Connect the cables to the node

You must connect all the power and network cables to the back of the node.
Procedure
1. Locate the labels on all the power and network cables to ensure that they are
connected correctly.

Power up the node

Power up the node by pressing the power button on the back panel.
Procedure
1. Power up the node by pressing the power button on the back panel of the node. It is
located just left of center, toward the upper part of the back panel.

Confirm a healthy node

Confirm that the node has powered up successfully.
Procedure
1. Review the node power-up for errors.
Do not continue until the node has successfully booted to the command prompt.

Note

If errors are present following a reboot, do not continue. Contact EMC Isilon Technical
Support.

Install the new drive

You install a drive in a node by inserting it into an empty drive bay.
Procedure
1. With the locking handle on the drive open, insert the drive into the empty drive bay by
sliding the drive along the bay rails until it stops.
Do not force the drive into the drive bay. Forcing the drive into the drive bay could
damage both the drive and the drive bay.

22 X400 Node Upgrade Guide

 Upgrade a node

2. Hold the drive in place and gently push the locking handle down against the end of
the drive to secure, or seat, the drive in the node.
Drives that are not fully seated will not be recognized when the node is started, and a
red light will appear above the drive. To avoid this problem, run your finger across all
the installed drives to ensure that they are all seated evenly.
3. Connect the drive to the node using the node number and bay number where you just
inserted the drive. Depending on your version of OneFS, run one of the following
commands:
OneFS 8.0 or later
isi devices drive add <bay-number> --node-lnn <node-number>

Earlier than OneFS 8.0

isi devices -a add -d <node-number>:<bay-number>

The add command connects the drive to the node.

If the add command fails, make sure that the drive is formatted by running one of the
following commands:
OneFS 8.0 or later
isi devices format <bay-number> --node-lnn <node-number>

Earlier than OneFS 8.0

isi devices -a format -d <node-number>:<bay-number>

The format command introduces the drive to the cluster.

If the drive does not respond to either of these commands, do not remove the drive
from the node. Contact Isilon Technical Support.

4. After the new drive has been installed, replace the front panel on the node by aligning
the panel with the front of the node and pressing the panel until it clicks onto the
node.
If you replaced a drive in the rear of the node, replace the EMI shield.

Run the post-upgrade script

Once you install the new hardware in the node, you must verify and report the upgrade by
running the post-upgrade script.
Procedure
1. To change to the correct directory, use the information in the name of your hardware
package to run the following command:

cd /var/crash/cto/hwPackages/IsiHW_Package_<order-number>_
<node-serial-number>_<date-time-stamp>

2. Verify installation of the hardware by running the following command:

./isi_cto_upgrade --post

Run the post-upgrade script 23

Node Upgrade Guide

Note

If the cluster is in SmartLock compliance mode, verify installation of the hardware by

running the following command:

sudo /usr/bin/isi_HWtools/isi_cto_upgrade --post

As the script confirms the upgrade, a number of status messages appear.

If an FTP connection is available, the script sends an updated ABR to Isilon Technical
Support.

Remove the upgrade package from the node

After you run all the scripts, it is safe to remove the upgrade package.
Procedure
1. Change out of the hardware package directory by typing the following command:
cd /

2. To delete the upgrade package from a node, use the information in the name of your
hardware package type the following command:
isi pkg delete --forced_local IsiHW_Package_ <order-number> _ <node-
serial-number> _ <date-time-stamp>

Update drive firmware

After you install a new drive in a node, update drive firmware.
This procedure explains how to update drive firmware on clusters running OneFS 7.1.1 or
later. If your cluster is running an earlier version of OneFS, you must download and install
the latest drive firmware package. For more information, see the latest drive firmware
package release notes available on https://support.emc.com/.

Note

Do not restart or power off nodes while drive firmware is being updated on the cluster.

Procedure
1. Open a secure shell (SSH) connection to any node in the cluster and log in.
2. Depending on your version of OneFS, run one of the following commands to update
the drive firmware for your cluster:
OneFS 8.0 or later
To update the drive firmware for your entire cluster, run the following command:
isi devices drive firmware update start all --node-lnn
all
To update the drive firmware for a specific node only, run the following
command:
isi devices drive firmware update start all --node-lnn
<node-number>

24 X400 Node Upgrade Guide

 Upgrade a node

OneFS 7.1.1 - OneFS 8.0

For OneFS versions between 7.1.1 - 8.0 you will need to run the following
command on each node that requires drive firmware:
isi devices -a fwupdate

CAUTION

You must wait for one node to finish updating before you initiate an update on the
next node. To confirm that a node has finished updating, run the following command:
isi devices -d <node-number>
A drive that is still updating will display a status of FWUPDATE.

Updating the drive firmware of a single drive takes approximately 15 seconds,

depending on the drive model. OneFS updates drives sequentially.

CONREF: Update drive firmware

The content in this topic is the source for conrefs related to the procedure for updating
drive firmware after you install a DSP.
Procedure
1. Open a secure shell (SSH) connection to any node in the cluster and log in.
2. Depending on your version of OneFS, run one of the following commands to update
the drive firmware for your cluster:
OneFS 8.0 or later
To update the drive firmware for your entire cluster, run the following command:
isi devices drive firmware update start all --node-lnn
all
To update the drive firmware for a specific node only, run the following
command:
isi devices drive firmware update start all --node-lnn
<node-number>

OneFS 7.1.1 - OneFS 8.0

For OneFS versions between 7.1.1 - 8.0 you will need to run the following
command on each node that requires drive firmware:
isi devices -a fwupdate

CAUTION

You must wait for one node to finish updating before you initiate an update on the
next node. To confirm that a node has finished updating, run the following command:
isi devices -d <node-number>
A drive that is still updating will display a status of FWUPDATE.

Updating the drive firmware of a single drive takes approximately 15 seconds,

depending on the drive model. OneFS updates drives sequentially.

CONREF: Update drive firmware 25

Node Upgrade Guide

Verify a drive firmware update

After you update the drive firmware in a node, confirm that the firmware is updated
properly and that the affected drives are operating correctly.
Procedure
1. Ensure that no drive firmware updates are currently in progress by running one of the
following commands:
OneFS 8.0 or later

isi devices drive firmware update list

Earlier than OneFS 8.0

isi devices

If a drive is currently being updated, [FW_UPDATE] appears in the status column.

2. Verify that all drives have been updated by running the following command:
OneFS 8.0 or later

isi devices drive firmware list

Earlier than OneFS 8.0

isi drivefirmware status

If all drives have been updated, the Desired FW column is empty.

3. Verify that all affected drives are operating in a healthy state by running the following
command:
OneFS 8.0 or later

isi devices drive list --node-lnn all

Earlier than OneFS 8.0

isi devices

If a drive is operating in a healthy state, [HEALTHY] appears in the status column.

Reviewing the upgrade

After you upgrade the required nodes, review the status of the cluster and the
configuration of all storage pools.

26 X400 Node Upgrade Guide

 Upgrade a node

Confirm new node configuration

Confirm that the nodes were upgraded successfully by viewing the current hardware
configuration.
Procedure
1. View the current hardware configuration, including installed RAM, by running the
following command:
isi_hw_status -iIc

2. View the installed drives by running the following command:

isi_radish -q

3. View the installed network interfaces by running the following command:

ifconfig | grep flags

4. View the current storage pools by running the following command:

isi status -d

Note

After you upgrade all the nodes in a node pool, the nodes should become part of a
new pool. It is possible that the old node pool will remain visible with no nodes in it. If
you see this issue, contact EMC Isilon Technical Support for assistance.

5. View the hardware as-built record by running the following command:

isi_inventory_tool --display --itemType=asbuilt

6. Confirm that the receipt was installed by running the following command:
isi_inventory_tool --display --itemType=receipt

7. Verify and upload the as-built record by running the following command:
isi_cto_update --abr

This command sends the as-built record report through SupportIQ.

Results
The output of these commands should match the new configure-to-order (CTO)
configuration.

Review SmartPools policies

Following an upgrade, it is recommended that you review SmartPools settings.
Procedure
1. Review all the SmartPools policies to ensure that they accommodate the new cluster
configuration.
For instructions on how to view and adjust file pool policies, see the OneFS CLI
Administration Guide.

Confirm new node configuration 27

Node Upgrade Guide

Review switch settings

Following a NIC upgrade, we recommend that you review switch settings.
Procedure
1. Review all switch settings to ensure that they accommodate the new NICs.
For instructions on how to configure a switch, refer to the manufacturer's
documentation.

Gather logs
After you complete maintenance on a cluster, gather cluster logs.
You must collect cluster logs after all maintenance. Cluster logs provide snapshots of the
cluster that you can review to make sure that maintenance is successful.
Procedure
1. Gather cluster logs by typing the command:
isi_gather_info

Update the install database

After all work is complete, update the install database.
Procedure
1. Browse to the EMC Product Registration and Install base Maintenance service portal,
at: http://emc.force.com/createPSCcase.
2. Select the Product Registration and Install Base Maintenance
option.
3. To open the form, select the IB Status Change option.
4. Complete the form with the applicable information.
5. To submit the form, click Submit.

Roll back a failed upgrade

If you encounter an issue during an upgrade engagement, you can roll back the upgrade
to the original hardware configuration.
If you need to roll back an upgrade, contact EMC Isilon Technical Support for assistance.

CAUTION

You can attempt to roll back an upgrade only if the upgrade package is still installed. If
you have already uninstalled the upgrade package, you cannot roll back the upgrade.

Procedure
1. Change directory to the location of the upgrade package by running the following
command:

cd /var/crash/cto/hwPackages/IsiHW_Package_<order-number>_
<node-serial-number>_<date-time-stamp>

28 X400 Node Upgrade Guide

 Upgrade a node

2. Run the following command:

./isi_cto_upgrade --rollback

Note

If the cluster is in SmartLock compliance mode, run the following command:

sudo /usr/bin/isi_HWtools/isi_cto_upgrade --rollback

The command output is similar to the following example:

Verifying package integrity

Package integrity is good
Node CTO capability verified
Chassis VPD verified
Node S/N verified
Importing receipt (./rollback_receipt.xml)
Success
Rollback receipt imported
HAL re-initialized
Unprovisioning drives, will get reprovisioned on reboot
...

At the end of the command output, you are prompted to power down the node.

3. Power down the node and replace the original hardware.

Where to go for support

Contact EMC Isilon Technical Support for any questions about EMC Isilon products.

Online Support Live Chat Create a Service Request

Telephone l United States: 1-800-SVC-4EMC (800-782-4362)
Support
l Canada: 800-543-4782
l Worldwide: +1-508-497-7901
l Local phone numbers for a specific country are available at EMC
Customer Support Centers

Help with Online For questions specific to EMC Online Support registration or access,
Support email support@emc.com.
Isilon Info Hubs For the list of Isilon info hubs, see the page on the EMC Isilon
Community Network, EMC Isilon Community Network. Isilon info
hubs organize Isilon documentation, videos, blogs, and user-
contributed content into topic areas, making it easy to find content
about subjects that interest you.

Support for IsilonSD Edge

If you are running a free version of IsilonSD Edge, community support is available through
the EMC Isilon Community Network. However, if you have purchased one or more licenses
of IsilonSD Edge, you can contact EMC Isilon Technical Support for assistance, provided
you have a valid support contract for the product.

Where to go for support 29

Node Upgrade Guide

Copyright © 2016 EMC Corporation. All rights reserved. Published in the USA.

Published September 2016

EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without
notice.

The information in this publication is provided as is. EMC Corporation makes no representations or warranties of any kind with
respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a
particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software
license.

EMC², EMC, and the EMC logo are registered trademarks or trademarks of EMC Corporation in the United States and other countries.
All other trademarks used herein are the property of their respective owners.

For the most up-to-date regulatory document for your product line, go to EMC Online Support (https://support.emc.com).

30 X400 Node Upgrade Guide