Isilon ® SolVe Generator

Solution for Validating your engagement

Topic
x200
Selections
Select Node Type: Non-CTO
X200 - Select Non-CTO Node Activity: Replace Boot Drive

Generated: April 13, 2020 9:20 PM GMT

REPORT PROBLEMS

If you find any errors in this procedure or have comments regarding this application, send email to
SolVeFeedback@emc.com
Copyright © 2019 Dell Inc. or its subsidiaries. All Rights Reserved.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” EMC CORPORATION (“EMC”)
MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE
INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-
INFRINGEMENT AND ANY WARRANTY ARISING BY STATUTE, OPERATION OF LAW, COURSE OF
DEALING OR PERFORMANCE OR USAGE OF TRADE. IN NO EVENT SHALL EMC BE LIABLE FOR
ANY DAMAGES WHATSOEVER INCLUDING DIRECT, INDIRECT, INCIDENTAL, CONSEQUENTIAL,
LOSS OF BUSINESS PROFITS OR SPECIAL DAMAGES, EVEN IF EMC HAS BEEN ADVISED OF THE
POSSIBILITY OF SUCH DAMAGES.

EMC believes the information in this publication is accurate as of its publication date. The information is
subject to change without notice. Use, copying, and distribution of any EMC software described in this
publication requires an applicable software license.

Dell, EMC, Dell EMC and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other
trademarks may be the property of their respective owners.

Publication Date: April, 2020

EMC Confidential Information version: 4.9.5.5

Page 1 of 23
 Contents
Preliminary Activity Tasks .......................................................................................................4
Read, understand, and perform these tasks.................................................................................................4

General Information for Removing and Installing FRUs..........................................................5

Avoiding Electrostatic Discharge (ESD) damage..........................................................................................5
Emergency procedures (without an ESD kit)...........................................................................................5
Removing, installing, or storing FRUs .....................................................................................................6
Boot Drive Replacement Guide...............................................................................................6
Isilon..............................................................................................................................................................6
120-0083-01 Rev H .................................................................................................................................6
Replace a boot drive ...............................................................................................................6
Replacing a boot drive ..................................................................................................................................6
Working with clusters in SmartLock compliance mode............................................................................7
Task 1: Gather logs.......................................................................................................................7
Task 2: Install a drive support package.........................................................................................8
Task 3: Download a Field Replacement Unit (FRU) package.......................................................8
Install the boot drive replacement .................................................................................................................9
Task 4: Write a sentinel file ...........................................................................................................9
Task 5: Identify a failed boot drive ................................................................................................9
Task 6: Power down the node.....................................................................................................11
Task 7: Slide the node out of the rack.........................................................................................12
Task 8: Remove the node top panel ...........................................................................................13
Task 9: Remove the cross bracket..............................................................................................13
Task 10: Replace the failed boot drive..........................................................................................14
Task 11: Install the cross bracket..................................................................................................16
Task 12: Install the node top panel ...............................................................................................16
Task 13: Return the node to the rack............................................................................................16
Task 14: Isolate the node from the cluster ....................................................................................16
Task 15: Power up the node .........................................................................................................17
Task 16: Verify sentinel file ...........................................................................................................17
Task 17: Verify healthy boot drives ...............................................................................................17
Task 18: Connect the node back to the cluster.............................................................................18
Install the FRU package and run scripts .....................................................................................................18
Task 19: Install the FRU package on the node .............................................................................18
Task 20: Run the update script .....................................................................................................19

EMC Confidential Information version: 4.9.5.5

Page 2 of 23
 Task 21: Run the ABR script.........................................................................................................19
Task 22: Generate an ABR ...........................................................................................................20
Task 23: Remove the FRU package from the node......................................................................20
Task 24: Update drive firmware ....................................................................................................20
Task 25: Verify a drive firmware update........................................................................................21
Task 26: Gather logs.....................................................................................................................22
Task 27: Returning a failed part to Isilon.......................................................................................22
Task 28: Update the install database............................................................................................22
Where to go for support ..............................................................................................................................22

EMC Confidential Information version: 4.9.5.5

Page 3 of 23
Preliminary Activity Tasks
This section may contain tasks that you must complete before performing this procedure.

Read, understand, and perform these tasks

1. [ ] Table 1 lists tasks, cautions, warnings, notes, and/or knowledgebase (KB) solutions that you
need to be aware of before performing this activity. Read, understand, and when necessary perform
any tasks contained in this table and any tasks contained in any associated knowledgebase solution.

Table 1 List of cautions, warnings, notes, and/or KB solutions related to this activity

2. [ ] This is a link to the top trending service topics. These topics may or not be related to this activity.
This is merely a proactive attempt to make you aware of any KB articles that may be associated with
this product.
Isilon Top Service Topics

EMC Confidential Information version: 4.9.5.5

Page 4 of 23
General Information for Removing and Installing FRUs
This section describes precautions you must take and general procedures you must follow when
removing, installing, or storing field-replaceable units (FRUs). The procedures in this section apply to FRU
handling during hardware upgrades as well as during general replacement.
FRUs are designed to be powered up at all times. This means you can accomplish FRU replacements
and most hardware upgrades while the cabinet is powered up. To maintain proper airflow for cooling and
to ensure EMI compliance, make sure all front bezels, filler panels, and filler modules are reinstalled after
the FRU replacement or hardware upgrade is completed.

Avoiding Electrostatic Discharge (ESD) damage

When replacing or installing FRUs, you can inadvertently damage the sensitive electronic circuits in the
equipment by simply touching them. Electrostatic charge that has accumulated on your body discharges
through the circuits. If the air in the work area is very dry, running a humidifier in the work area helps
decrease the risk of ESD damage. You must use the following procedures to prevent damage to the
equipment, so read the following instructions carefully.
 Provide enough room to work on the equipment. Clear the work site of any unnecessary materials or
materials that naturally build up electrostatic charge, such as foam packaging, foam cups, cellophane
wrappers, and similar items.
 Do not remove replacement or upgrade FRUs from their antistatic packaging until you are ready to
install them.
 Gather the ESD kit and all other materials you will need before you service a component. Once
servicing begins, you should avoid moving away from the work site; otherwise, you may build up an
electrostatic charge.
 Use the ESD kit when handling any FRU.
If an emergency arises and the ESD kit is not available, follow the procedures in the section
Emergency procedures (without an ESD kit).
 Use the ESD wristband that is supplied with your system.
To use it, attach the clip of the ESD wristband (strap) to any bare (unpainted) metal on the cabinet
enclosure; then secure the wristband around your wrist with the metal button against your skin.

Emergency procedures (without an ESD kit)

In an emergency, when an ESD kit is not available, use the following procedures to reduce the possibility
of an electrostatic discharge by ensuring that your body and the subassembly are at the same
electrostatic potential.

IMPORTANT: These procedures are not a substitute for the use of an ESD kit. You should follow them
only in the event of an emergency.

 Before touching any FRU, touch a bare (unpainted) metal surface of the enclosure.
 Before removing any FRU from its antistatic bag, place one hand firmly on a bare metal surface of the
enclosure, and at the same time, pick up the FRU while it is still sealed in the antistatic bag. Once you
have done this, do not move around the room or contact other furnishings, personnel, or surfaces
until you have installed the FRU.
 When you remove a FRU from the antistatic bag, avoid touching any electronic components and
circuits on it.

EMC Confidential Information version: 4.9.5.5

Page 5 of 23
  If you must move around the room or touch other surfaces before installing a FRU, first place the
FRU back in the antistatic bag. When you are ready again to install the FRU, repeat these
procedures.

Removing, installing, or storing FRUs

Use the following precautions when you remove, handle, or store FRUs:
 Do not remove a FRU from the cabinet until you have the replacement available.
 Handle a FRU only when wearing a properly attached ESD wristband:
Attach the clip of the ESD wristband to the ESD bracket or to a bare metal portion of the FRU’s
enclosure.
Secure the wristband around your wrist with the metal button against your skin.
 Handle a FRU gently. A sudden jar, drop, or violent vibration can permanently damage some FRUs.
 The weight of some FRUs is not evenly distributed. To prevent personal injury or equipment damage
when removing, installing, or handling theses FRUs, use both hands and provide extra support at the
heavy end of the FRU.
 Never use excessive force to remove or install a FRU.
 Store a FRU in the antistatic bag and the specially designed shipping container in which you received
it. Use this special shipping container when you need to return the FRU.
 If you need to store a FRU for a short time, make sure the temperature of the location where you
store the FRU is within the limits specified by the FRU’s Technical Specifications.
 Disk modules are sensitive to the extreme temperatures sometimes encountered during shipping. We
recommend that you leave a new disk module in its shipping container and allow it to acclimate to the
installation site’s ambient temperature for at least four hours before operating the new module in the
system.
 When removing a disk module, pull the module part way out of the slot; then wait 30 seconds for the
drive to spin down before removing it.
 When installing multiple disks in a powered-up system, wait at least 6 seconds before sliding the
second and each subsequent disk into the system.
 Place a disk module on a soft, antistatic surface, such as an industry-standard antistatic foam pad or
the container used to ship the module. Never place a disk module directly on a hard surface.



Boot Drive Replacement Guide

Isilon
120-0083-01 Rev H

Replace a boot drive

Replacing a boot drive
You can replace a failed boot drive in the field.

EMC Confidential Information version: 4.9.5.5

Page 6 of 23
This node contains two flash boot drives. The boot drives contain data vital to the health of the node,
including the OneFS operating system and backups of the node journal. The two boot drives mirror each
other, which provides a backup source of data if one of the drives fails.
You must identify which of the two boot drives has failed before you shut down the node. If you
accidentally remove the healthy drive, the data on the node can be lost.

CAUTION: If this procedure is not followed accurately, data loss and severe disruption of cluster
operations can occur. Perform every step in this procedure; if the system does not respond as expected,
contact Isilon Technical Support.

CAUTION: Perform this procedure on only one node at a time. Performing maintenance on multiple
nodes in parallel may lower the protection level of the cluster, put data at risk, and lead to the interruption
of client workflows.

Working with clusters in SmartLock compliance mode

Clusters running in SmartLock compliance mode require a sudo prefix to run root commands.
If a cluster is running in SmartLock compliance mode, root access is disabled on the cluster. Because of
this, you can run some commands only through the sudo program. Prefixing a command with sudo
enables you to run commands that require root access. For example, if you do not have root access, the
following command fails:
isi drivefirmware status
However, if you are on the sudoers list, the following command succeeds:
sudo isi drivefirmware status
Compliance mode commands that require changes beyond the sudo prefix are noted in the procedure
steps.
For more information on the sudo program and compliance mode commands, see the OneFS CLI
Administration Guide.

Task 1: Gather logs

Before you begin any maintenance on a cluster, gather cluster logs.
About this task
You must collect cluster logs before all maintenance procedures. Cluster logs provide snapshots of the
cluster, which you can review to make sure that maintenance is successful.
Procedure
1. [ ] Open a secure shell (SSH) connection to any node in the cluster and log in.
2. [ ] Gather cluster logs by running the following command:
isi_gather_info

EMC Confidential Information version: 4.9.5.5

Page 7 of 23
Task 2: Install a drive support package
Before you install a new drive, download and install the drive support package. The drive support
package updates drive configuration information on the node, and contains the latest firmware versions
for Isilon qualified drives.
About this task
The drive support package is only supported by clusters running OneFS 7.1.1 or later. If your cluster is
running an earlier version of OneFS, you can skip this step.
Procedure
1. [ ] Go to the EMC Support page that lists all the available versions of the drive support package.
2. [ ] Click the latest version of the drive support package and download the file.

Note: See the Considerations for installing the latest drive support package section in order to select
the appropriate variant of the package. If you are unable to download the package, contact EMC
Isilon Technical Support for assistance.

3. [ ] Open a secure shell (SSH) connection to any node in the cluster and log in.
4. [ ] Create or check for the availability of the directory structure
/ifs/data/Isilon_Support/dsp.
5. [ ] Copy the downloaded file to the dsp directory through SCP, FTP, SMB, NFS, or any other
supported data-access protocols.
6. [ ] Unpack the drive support package by running the following command:
tar -zxvf Drive_Support_<version>.tgz
7. [ ] Install the package by running the following command:
isi_dsp_install Drive_Support_<version>.tar

Note: You must run the isi_dsp_install command to install the dirve support package. Do not
use the isi pkg command.

Task 3: Download a Field Replacement Unit (FRU) package

Before you replace a component in a configure-to-order (CTO) node, obtain a Field Replacement Unit
(FRU) package from the EMC FTP site. The FRU package updates the CTO and as-built information on
the node, then forwards the updated information to Isilon Technical Support.
About this task
Note: It is not necessary to obtain a FRU package for non-CTO nodes. If you are performing
maintenance on a non-CTO node, skip all steps related to the FRU package.

Procedure
1. [ ] Download the latest FRU package from ftp://ftp.emc.com/outgoing/Fru_Package/.
2. [ ] Note the name of the FRU package. You will use the name for other commands.
Package names follow this convention:
IsiFru_Package_ <date-time-stamp> .tgz
For example: IsiFru_Package_201507072125.tgz

EMC Confidential Information version: 4.9.5.5

Page 8 of 23
 3. [ ] Place the FRU package on the cluster through a network drop, or by asking someone at the
cluster site to place the package for you. If neither of these options is available to you, contact Isilon
Technical Support for assistance.

Install the boot drive replacement

Identify and remove the failed boot drive, and then install and test the replacement drive.

Task 4: Write a sentinel file

Because replacement of an incorrect boot drive severely affects cluster stability, verify the successful
replacement of the failed drive. Prior to replacing the failed drive, write a sentinel file to the root partition.
After the failed drive is replaced, the file is only readable if the correct boot drive is replaced.
Procedure
1. [ ] Use the output of isi devices as the content of the file. Write a sentinel file to the root partition by
typing the command:
isi devices > /sentinel.txt

Task 5: Identify a failed boot drive

When performing a boot drive replacement on a node with two boot drives, first determine which of the
two drives must be replaced.
Procedure
1. [ ] Using a serial cable, connect to the node you are going to work on.
2. [ ] View boot drive information. Depending on your version of OneFS, run one of the following
commands:
OneFS 8.0 or later
camcontrol devlist | grep ad
The following information appears:
<SanDisk SSD P4 8GB SSD 8.10> at scbus1 target 0 lun 0 (pass13,ada0)
<SanDisk SSD P4 8GB SSD 8.10> at scbus2 target 1 lun 0 (pass14,ada1)

The boot drives are listed in the left column. In the previous example, both boot drives are healthy. If
one of the boot drives has failed, the drive will not appear in the output.
Make note of whether the failed boot drive is the ada1 or ada0 device, and then use the following
table to determine the location of the boot drive inside the node.

Boot Order OneFS Drive ID Board Drive Slot Inside Node

Master ada1 J4
Slave ada0 J3

Make note of the board drive slot that contains the failed boot drive. The ada1 drive is on the right
side of the boot carrier card. The ada0 drive is on the left side of the boot carrier card.

CAUTION: If both drives appear to have failed, do not continue. Contact Isilon Technical Support
immediately.

EMC Confidential Information version: 4.9.5.5

Page 9 of 23
Earlier than OneFS 8.0
atacontrol list
The following information appears:
ATA channel 0:
Master: no device present
Slave: no device present
ATA channel 1:
Master: no device present
Slave: no device present
ATA channel 2:
Master: ad4 <SanDisk SSD P4 8GB/SSD 8.10> Serial ATA v1.0 II
Slave: no device present
ATA channel 3:
Master: no device present
Slave: ad7 <SanDisk SSD P4 8GB/SSD 8.10> Serial ATA v1.0 II
ATA channel 4:
Master: no device present
Slave: no device present
ATA channel 5:
Master: no device present
Slave: no device present

The boot drives are listed under ATA channel 2 (master) and ATA channel 3 (slave). In the
previous example, both boot drives are healthy. If one of the boot drives has failed, the display reads
no device present for that drive.
Make note of whether the failed boot drive is the ad4 or ad7 device, and then use the following table
to determine the location of the boot drive inside the node.

Boot Order OneFS Drive ID Board Drive Slot Inside Node

Master ad4 J3
Slave ad7 J4

Make note of the board drive slot that contains the failed boot drive.

CAUTION: If both drives appear to have failed, do not continue. Contact Isilon Product Support
immediately.

3. [ ] If both drives appear to be healthy, one of the drives may have partially failed. To identify a
partially failed drive, check the status of the individual partition mirrors by typing the following
command:
gmirror status
From left to right, the output displays the name of each mirror, the status of the mirror relationship,
and the component IDs for each boot drive.
The following example shows the boot drive partition layout in a healthy node. The mirrors for each
partition show:
 a value of COMPLETE in the Status column.

EMC Confidential Information version: 4.9.5.5

Page 10 of 23
  the component IDs for both boot drives in the Components column. The component IDs are a
combination of the OneFS Drive ID, and the partition number (the number following the letter p).
Both boot drives are listed for each mirror with the exception of the var-crash mirror, which only
lists the slave drive.

Note: If you are running OneFS 8.0 or later, your OneFS Drive IDs will display as ada0 or ada1.The
partition numbers in the display may differ from the following example.

Name Status Components

mirror/root0 COMPLETE ada7p4
ada4p4
mirror/var-crash COMPLETE ada7p10
mirror/mfg COMPLETE ada7p9
ada4p10
mirror/journal-backup COMPLETE ada7p8
ada4p8
mirror/var1 COMPLETE ada7p7
ada4p7
mirror/var0 COMPLETE ada7p6
ada4p6
mirror/root1 COMPLETE ada7p5
ada4p5

The following example shows the boot drive partition layout as it appears in the event of a failed boot
drive. A failed boot drive forces the mirrors for a partition to show:
 a value of DEGRADED in the Status column.
 only the component ID of the healthy boot drive in the Components column. The failed boot drive
does not appear.

Note: DEGRADED does not refer to a specific drive, but to the mirror relationship between the drives.
If a drive appears in the Components column next to the DEGRADED status, it is healthy and should
not be removed.

Name Status Components

mirror/root0 DEGRADED ad4p4
mirror/var-crash COMPLETE ad7p10
mirror/mfg COMPLETE ad7p9
ad4p10
mirror/journal-backup COMPLETE ad7p8
ad4p8
mirror/var1 COMPLETE ad7p7
ad4p7
mirror/var0 DEGRADED ad4p6
mirror/root1 COMPLETE ad7p5
ad4p5

In the previous example, ad7p4 is missing from the degraded partition mirror/root0, and ad7p6 is
missing from the degraded partition mirror/var0. The missing drive, ad7, is the failed drive.
Determine which drive has failed. Use the previous table to determine which board drive slot contains
the failed boot drive and make a note of the number (J3 or J4).

Note: If both drives have failed, do not continue. Contact Isilon Product Support.

Task 6: Power down the node

Power down the node before performing maintenance.

EMC Confidential Information version: 4.9.5.5

Page 11 of 23
 Procedure
1. [ ] Connect to an available node in the cluster with a serial cable or network drop.
2. [ ] Determine the IP address of the node you are powering down by typing the command:
isi status -q
3. [ ] From the node that you connected to, open a secure shell (SSH) connection to the node that is
to be shut down by typing the command:
ssh <node_ip_address>
4. [ ] Power down the node by typing the following command:
shutdown -p now
If the node does not respond to the shutdown command, press the Power button on the node three
times, and then wait five minutes. If the node still does not shut down, you are at risk for losing data.
Do not proceed. Contact EMC Isilon Technical support for assistance.

CAUTION: A forced power down should be attempted only if a node is unresponsive. Forcing the
power down of a healthy node can result in data loss.

5. [ ] Verify that the node is powered down by typing the following command:
isi status -q
Confirm that the node has a status of D--R (Down, Read Only). See node 3 in the following example.
ID |IP Address |DASR| In Out Total| Used / Size |Used / Size
---+---------------+----+-----+-----+-----+------------------+-
1|10.53.217.201 | OK | 48M| 0| 48M| 19G/ 6.2T(< 1%)|(No SSDs)
2|10.53.217.202 | OK | 46M| 0| 46M| 23G/ 6.2T(< 1%)|(No SSDs)
3|10.53.217.203 |D--R| n/a| n/a| n/a| n/a/ n/a( n/a)|n/a/n/a( n/a)

Task 7: Slide the node out of the rack

Slide the node away from the rack to access the contents of the node.
Procedure
1. [ ] Label the InfiniBand, ethernet, and power cables connected to the back of the node to ensure
that they are reconnected correctly.
2. [ ] Disconnect all cables from the back of the node.

Note: If there are transceivers connected to the end of your IB or ethernet cables, make sure to
remove them with the cables. If you are using fiber ethernet cables, you will need to disconnect the
cable from the transceiver, then remove the transceiver from the node.

3. [ ] Remove the node front panel.

4. [ ] Remove the retaining screws that secure the node to the rack cabinet.
5. [ ] Slide the node from the rack cabinet to fully extend the slide rails and provide clear access to the
node. Do not remove the node from the slide rails.

EMC Confidential Information version: 4.9.5.5

Page 12 of 23
 DANGER: Slide the node out from the rack slowly. Do not extend the rails completely until you
confirm that the node is latched and safely secured to the rails.

Task 8: Remove the node top panel

You remove the top panel to gain access to the contents of the node.
About this task

WARNING: Properly ground yourself to prevent electrostatic discharge from damaging the node. For
example, attach an ESD strap to your wrist and the node chassis.

Procedure
1. [ ] Loosen the captive screw that secures the node top panel.
2. [ ] Slide the top panel toward the rear of the node, and then lift the top panel to access the node
interior.

Task 9: Remove the cross bracket

Removing the cross bracket provides clear access to the inside of the node.
Procedure
1. [ ] Locate the cross bracket within the node.

EMC Confidential Information version: 4.9.5.5

Page 13 of 23
 Figure 1 Cross bracket

2. [ ] Remove the cross bracket by pressing on both sides of the node chassis where the cross
bracket is connected, then lift the bracket out of the node.

Task 10: Replace the failed boot drive

After the node is open, locate the failed boot drive. Remove and replace the failed drive with a new drive.
About this task

EMC Confidential Information version: 4.9.5.5

Page 14 of 23
 1. Boot drive

Procedure
1. [ ] Locate the two board drive slots that contain the boot drives. The slots are labeled J3 and J4.
Gently pull the failed boot drive from the board drive slot.

EMC Confidential Information version: 4.9.5.5

Page 15 of 23
 1. J3 connector 2. J4 connector

2. [ ] Insert the replacement boot drive into the empty boot drive slot and gently press down to secure
the drive.

Task 11: Install the cross bracket

You must install the cross bracket by hooking it to the bracket holes on the interior of the node and then
snapping the other end into the chassis wall.
About this task

WARNING: The cross bracket sits directly above the boot drives. Use caution when installing the cross
bracket so that the boot drives are not dislodged or damaged.

Task 12: Install the node top panel

You must secure the top panel onto the node.
Procedure
1. [ ] Place the top panel on the node so that the front edge of the top panel is about one inch behind
the drive bays, and then slide the top panel forward into place.

WARNING: The chassis intrusion switch can be damaged if the top panel is slid too far back on the
node.
2. [ ] Tighten the captive top panel screw to secure the top panel to the node.

Task 13: Return the node to the rack

Return the node to the rack after all work is complete.
Procedure
1. [ ] Slide the node back into the rack cabinet.

WARNING: Slide the node slowly so you do not slam the node into the rack and damage the node.
2. [ ] Reconnect the ethernet, InfiniBand, and power cables to the back of the node.
3. [ ] Secure the node to the rack cabinet.
4. [ ] Replace the node front panel.

Task 14: Isolate the node from the cluster

Isolate the node from the cluster by disconnecting it from the InfiniBand network.
Procedure
1. [ ] Label the InfiniBand cables to ensure that they are reconnected properly later.

EMC Confidential Information version: 4.9.5.5

Page 16 of 23
 2. [ ] Disconnect the InfiniBand cables from the back of the node.
3. [ ] Connect directly to the node using a serial cable.

Task 15: Power up the node

Power up the node by pressing the power button on the back panel.
Procedure
1. [ ] Power up the node by pressing the power button on the back panel of the node. It is located in
the upper-left corner of the back panel.

Task 16: Verify sentinel file

Confirm that the correct boot drive is replaced. Locate the sentinel file that was sent to the root partition.
Procedure
1. [ ] Locate the sentinel file in the root partition by typing the following command:
cat /sentinel.txt
If the sentinel file appears, you replaced the correct boot drive. If the file is missing, do not continue.
Contact Isilon Product Support.
2. [ ] Remove the file by typing the following command:
rm /sentinel.txt

Task 17: Verify healthy boot drives

Verify that the new drive is active and healthy after the replacement boot drive is installed in the node.
Procedure
1. [ ] Verify that the boot drives are healthy by typing the following command:
gmirror status
The following information appears:

Note: If you are running OneFS 8.0 or later, your OneFS Drive IDs will display as ada0 or ada1.The
partition numbers in the display may differ from the following example.

Name Status Components

mirror/root0 COMPLETE ad7p4
ad4p4
mirror/var-crash COMPLETE ad7p10
mirror/mfg COMPLETE ad7p9
ad4p10
mirror/journal-backup COMPLETE ad7p8
ad4p8
mirror/var1 COMPLETE ad7p7
ad4p7
mirror/var0 COMPLETE ad7p6
ad4p6
mirror/root1 COMPLETE ad7p5
ad4p5

Confirm that the values in the Status column all read COMPLETE.
2. [ ] Verify boot drive information. Depending on your version of OneFS, run one of the following
commands:
OneFS 8.0 or later

EMC Confidential Information version: 4.9.5.5

Page 17 of 23
 camcontrol devlist | grep ad
The following information appears:
<SanDisk SSD P4 8GB SSD 8.10> at scbus1 target 0 lun 0 (pass13,ada0)
<SanDisk SSD P4 8GB SSD 8.10> at scbus2 target 1 lun 0 (pass14,ada1)

Earlier than OneFS 8.0

atacontrol list
The following information appears:
ATA channel 0:
Master: no device present
Slave: no device present
ATA channel 1:
Master: no device present
Slave: no device present
ATA channel 2:
Master: ad4 <SanDisk SSD P4 8GB/SSD 8.10> Serial ATA v1.0 II
Slave: no device present
ATA channel 3:
Master: no device present
Slave: ad7 <SanDisk SSD P4 8GB/SSD 8.10> Serial ATA v1.0 II
ATA channel 4:
Master: no device present
Slave: no device present
ATA channel 5:
Master: no device present
Slave: no device present

Ensure that both boot drives are listed.

Task 18: Connect the node back to the cluster

Reconnect all InfiniBand cables to add the node back into the cluster.
Procedure
1. [ ] Connect all InfiniBand cables to the rear of the node.
The node merges with the cluster; it is unnecessary to power cycle the node.

Install the FRU package and run scripts

Update the configure-to-order (CTO) and as-built information on the node by installing a FRU package.

Note: It is not necessary to update CTO and as-built information on non-CTO nodes. If you are
completing maintenance on a non-CTO node, skip all steps related to the FRU package.
Note: If your cluster is running in SmartLock compliance mode with OneFS 7.0.2.10 or later, 7.0.1.4 or
later, or 7.1.1.0 or later you will need to enter the provided compliance mode commands to run the FRU
scripts. If your cluster is running in compliance mode but is not running one of these versions, you will
need to upgrade your OneFS version to support the compliance mode commands. Contact Isilon
Technical Support.

Task 19: Install the FRU package on the node

Unpack and install the FRU package on the node.
Procedure
1. [ ] Place the FRU package on the node.

EMC Confidential Information version: 4.9.5.5

Page 18 of 23
 2. [ ] Unpack the FRU package by running the following command:
tar -zxvf IsiFru_Package_<date-time-stamp>.tgz
3. [ ] Type cd to change to the directory containing the FRU tar.
4. [ ] Install the package. Depending on your version of OneFS, run one of the following commands:
OneFS 8.0 or later
isi upgrade patches install IsiFru_Package_<date-time-stamp>_.tar
Earlier than OneFS 8.0
isi pkg install IsiFru_Package_<date-time-stamp>.tar
As the package installs, the following message appears:
Preparing to install the package...
Checking the package for installation...
Installing the package
Committing the installation...
Package is committed.

Task 20: Run the update script

After the FRU package is installed on the node, run the update script.
Procedure
1. [ ] Move to the FRU package location by running the following command:
cd /var/crash/cto/fruPackages/IsiFru_Package_<date-time-stamp>
2. [ ] Perform the update script by running the following command:
./isi_fru_update_cluster
The system displays confirmation of the following items:
 CTO capability
 Current node hardware configuration

Task 21: Run the ABR script

Run the As Built Record (ABR) script to report the updated hardware to Isilon Technical Support.
Procedure
1. [ ] Verify installation of the updated hardware by running the following command:
./isi_cto_update --abr
The update is verified and a series of status messages confirm the node configuration, and if an FTP
connection is available, an updated ABR is sent to Isilon Technical Support.
2. [ ] If an external connection is not available, manually collect and deliver to Isilon Technical Support
the updated ABR.
3. [ ] If the cluster is running in SmartLock compliance mode, verify installation of the updated
hardware by running the following command:
sudo /usr/bin/isi_hwtools/isi_cto_update --abr --filepath .

Note: You must include the period at the end of the command.

EMC Confidential Information version: 4.9.5.5

Page 19 of 23
Sending an ABR to Isilon with no connectivity
If no external connectivity is available, the As Built Record on a Configure to Order (CTO) node cannot be
automatically delivered to Isilon Technical Support.
If external connectivity is available, the ABR is automatically generated and delivered to Isilon Technical
Support. If there is no external connectivity available, you must generate and copy the ABR from the
node, and then send the ABR to Isilon Technical Support through an alternate connection.

Task 22: Generate an ABR

You can manually send an As Built Record (ABR) by copying an XML file from the node and emailing the
file to Isilon Technical Support. You need network access to the node, or you can request that the
customer provide the file to you.
Procedure
1. [ ] Generate an ABR by running the following command:
isi_make_abr
The command generates a temporary file named asbuilt_ <serial-number>_<date-time-
stamp> .xml.
2. [ ] Identify the full name of the ABR file by running the following command:
isi_inventory_tool --display --itemType asbuilt | grep asbuiltFileName=
The system output contains information about the ABR file.
3. [ ] Place the ABR file where you can copy it by running the following command:
isi_inventory_tool --display --itemType asbuilt > /ifs/asbuilt_ <serial-
number>_<date-time-stamp> .xml
4. [ ] Copy the generated asbuilt_ <serial-number>_<date-time-stamp> .xml file.
5. [ ] If an FTP connection is not available, contact Isilon Technical Support for an alternate delivery
method.

Task 23: Remove the FRU package from the node

After all scripts are run, remove the FRU package from the node.
Procedure
1. [ ] Change out of the FRU package directory by running the following command:
cd /
2. [ ] Delete the FRU package from the node. Depending on your version of OneFS, run one of the
following commands:
OneFS 8.0 or later
isi upgrade patches uninstall IsiFru_Package_ <date-time-stamp>
Earlier than OneFS 8.0
isi pkg delete IsiFru_Package_ <date-time-stamp>

Task 24: Update drive firmware

After you install a new drive in a node, update drive firmware.
About this task

EMC Confidential Information version: 4.9.5.5

Page 20 of 23
This procedure explains how to update drive firmware on clusters running OneFS 7.1.1 or later. If your
cluster is running an earlier version of OneFS, you must download and install the latest drive firmware
package. For more information, see the latest drive firmware package release notes available on
https://support.emc.com/.

Note: Do not restart or power off nodes while drive firmware is being updated on the cluster.

Procedure
1. [ ] Open a secure shell (SSH) connection to any node in the cluster and log in.
2. [ ] Depending on your version of OneFS, run one of the following commands to update the drive
firmware for your cluster:
OneFS 8.0 or later
To update the drive firmware for your entire cluster, run the following command:
isi devices drive firmware update start all --node-lnn all
To update the drive firmware for a specific node only, run the following command:
isi devices drive firmware update start all --node-lnn <node-number>
OneFS 7.1.1 - OneFS 8.0
For OneFS versions between 7.1.1 - 8.0 you will need to run the following command on each node
that requires drive firmware:
isi devices -a fwupdate

CAUTION: You must wait for one node to finish updating before you initiate an update on the next
node. To confirm that a node has finished updating, run the following command:
CAUTION: isi devices -d <node-number>

CAUTION: A drive that is still updating will display a status of FWUPDATE.

Updating the drive firmware of a single drive takes approximately 15 seconds, depending on the drive
model. OneFS updates drives sequentially.

Task 25: Verify a drive firmware update

After you update the drive firmware in a node, confirm that the firmware is updated properly and that the
affected drives are operating correctly.
Procedure
1. [ ] Ensure that no drive firmware updates are currently in progress by running one of the following
commands:
OneFS 8.0 or later
isi devices drive firmware update list
Earlier than OneFS 8.0
isi devices
If a drive is currently being updated, [FW_UPDATE] appears in the status column.
2. [ ] Verify that all drives have been updated by running the following command:

EMC Confidential Information version: 4.9.5.5

Page 21 of 23
OneFS 8.0 or later
isi devices drive firmware list
Earlier than OneFS 8.0
isi drivefirmware status
If all drives have been updated, the Desired FW column is empty.
3. [ ] Verify that all affected drives are operating in a healthy state by running the following command:
OneFS 8.0 or later
isi devices drive drive list --node-lnn all
Earlier than OneFS 8.0
isi devices
If a drive is operating in a healthy state, [HEALTHY] appears in the status column.

Task 26: Gather logs

After you complete maintenance on a cluster, gather cluster logs.
About this task
You must collect cluster logs after all maintenance. Cluster logs provide snapshots of the cluster that you
can review to make sure that maintenance is successful.
Procedure
1. [ ] Gather cluster logs by typing the command:
isi_gather_info

Task 27: Returning a failed part to Isilon

Return the failed part to Isilon Technical Support.
Procedure
1. [ ] Contact Isilon Technical Support to notify them that you are returning a failed part.
2. [ ] Package the failed part in the packaging materials provided with the replacement part.
3. [ ] Attach the return label that was included with the replacement part.
4. [ ] For the RMA number, write the support case number provided by Isilon Technical Support.
5. [ ] Ship the failed part to the address specified on the return label.

Task 28: Update the install database

After all work is complete, update the install database.
Procedure
1. [ ] Browse to the EMC Product Registration and Install base Maintenance service portal, at:
http://emc.force.com/createPSCcase.
2. [ ] Select the Product Registration and Install Base Maintenance option.
3. [ ] To open the form, select the IB Status Change option.
4. [ ] Complete the form with the applicable information.
5. [ ] To submit the form, click Submit.

Where to go for support

Contact EMC Isilon Technical Support for any questions about EMC Isilon products.

EMC Confidential Information version: 4.9.5.5

Page 22 of 23
 Online Support Live Chat
Create a Service Request
Telephone Support United States: 1-800-SVC-4EMC (800-782-4362)
Canada: 800-543-4782
Worldwide: +1-508-497-7901
For local phone numbers for a specific country, see
EMC Customer Support Centers.
Help with Online Support For questions specific to EMC Online Support
registration or access, email support@emc.com.
Isilon Info Hubs For the list of Isilon info hubs, see the Isilon Info Hubs
page on the EMC Isilon Community Network. Isilon info
hubs organize Isilon documentation, videos, blogs, and
user-contributed content into topic areas, making it easy
to find content about subjects that interest you.

Support for IsilonSD Edge

If you are running a free version of IsilonSD Edge, community support is available through the EMC Isilon
Community Network. However, if you have purchased one or more licenses of IsilonSD Edge, you can
contact EMC Isilon Technical Support for assistance, provided you have a valid support contract for the
product.

EMC Confidential Information version: 4.9.5.5

Page 23 of 23