Avoiding Data Center Disasters

Keith Klesner, PE SVP, North America

Eric Maddison CEng Senior Consultant, EMEA

May 2017

© 2015 Uptime Institute, LLC

 1000+ Certifications in over 70
Countries

1 © 2015 Uptime Institute, LLC

 Success of a data center project
• Business Objectives or Mission Imperatives drive
design (functionality, capacity, capability) of a data
center
— Tier Standard: Topology
• Operations management and building traits to avoid
downtime are the measure of return on investment
— Tier Standard: Operational Sustainability

http://uptimeinstitute.com/publications
2 © 2015 Uptime Institute, LLC
 Why Uptime Institute Tiers?
• Internationally recognized de facto standard
— To date, Uptime Institute has awarded nearly 170 Tier Certifications
of Constructed Facilities in 43 countries
— Separated from TIA-942 in March 2014
• Only data center benchmarking system developed by and for
data center owners
• Rate a single site, compare two sites, or evaluate a portfolio
• Performance-based fundamental concepts
— Not a checklist, design menu, or cookbook
— Encourages innovative engineering solutions
— Allows for client equipment preferences

3
3 © 2015 Uptime Institute, LLC
 Tier Topology Categories
• Tier Classifications represent design topology
concepts meaningful in IT
— Tier I: Basic Capacity
— Tier II: Redundancy
— Tier III: Concurrent Maintenance
— Tier IV: Fault Tolerance
• Fractional “concepts” are not rationalized
— No Standard for Tier III.6
— Tier III + is undefined
— Data center’s Tier Classification based on lowest system
rating

4 © 2015 Uptime Institute, LLC

 Operational Sustainability
• Complements Tier Standard: Topology
• Differentiates the value of investment within a Tier
level
• Focuses on the effectiveness of investment over the
life of the facility
• Aggregates Positive and Negative impact of design
and management choices
— Management & Operations processes
— Built environment
— Site selection

5 © 2015 Uptime Institute, LLC

 Tier Certification
• Tier Certification of Design Documents
— In-depth review of design to ensure full compliance
• Tier Certification of Constructed Facility
— On-site verification of installed infrastructure
• Tier Certification of Operational Sustainability
— On-site verification of ongoing operations

Uptime Institute also offers preliminary and

progress reviews of designs and operations

6 © 2015 Uptime Institute, LLC

 Data Center Projects—A longstanding
opportunity for improvement
• Since the 1990s, Uptime Institute has been counseling the industry on
missteps during the planning and justification stage of a data center
• Industry feedback gathered through the Uptime Institute Network &
other owners/operators
• As the ATD program grew to over 1,000 elite engineers, became
fluent in the frustrations of the data center design community
• As Tier Certifications expanded, gained insight into hundreds of
projects globally

7 © 2015 Uptime Institute, LLC

 Designers, Owners, and Operators
must be able to answer:
• Is the design characteristic a proven
performance enhancement?
• Does the building characteristic decrease exposure
to human error?
• Do my operations teams and vendors have the
organization, training and tools to do their job?

8 © 2015 Uptime Institute, LLC

Design, Construction and Operations
Misses

© 2015 Uptime Institute, LLC

 Common Commissioning (Cx)
shortfalls
• Construction delays push project completion into
planned Level 4 and Level 5 Cx schedules
• Lack of independent Cx agent “self certification”
• Not operating on engine generators
• Cx testing regimen
— Not commissioning to design intent/OPR
— Focusing on fault scenarios versus maintenance

10 © 2015 Uptime Institute, LLC

 Not all disasters are immediate
• Value engineering often seen as positive in
commercial construction can have serious impacts
years later
• Misunderstanding by inexperienced construction
teams
— Do not understand Concurrent Maintainability or Fault
Tolerance
— Focused on getting it done versus a mission critical
mentality

11 © 2015 Uptime Institute, LLC

 Engine-generator controls

• Designed as N+2, Commissioned as N+1

• On isolation of the 2nd redundant engine generator
— Control system interpreted as a fault condition
— Opened all output breakers, disabling all engine-generator
power
• What was the operations plan?

12 © 2015 Uptime Institute, LLC

 Not accessible
• Systems was
incorrectly installed
• Cx team had to crawl on
and over to operate
• Operational Sustainability?

13 © 2015 Uptime Institute, LLC

 Critical makeup water system?

14 © 2015 Uptime Institute, LLC

 Material substitutions

15 © 2015 Uptime Institute, LLC

 Staffing and Organization Significant
Findings
• Staffing
— Inadequate staffing
— Excessive overtime (over 10%)
— No escalation process
• Qualification
— No list of required qualifications
— No experience with data center specific equipment
• Organization
— Roles and Responsibilities not documented
— Data center organization not integrated

16 © 2015 Uptime Institute, LLC

 Maintenance Significant Findings
• Preventive Maintenance (PM)
— No list of required PM activities
— PM activities not fully scripted
— No quality control process
• Housekeeping
— Combustibles in the data center
— No documented housekeeping policy
• Maintenance Management System (MMS)
— No list of equipment
— Missing critical data: warranty info, maintenance history,
performance data, etc.

17 © 2015 Uptime Institute, LLC

 Training Significant Findings
• Data Center Staff
— Undocumented On-the-Job (OJT) programs
— No formal qualification program
— No list of training required by position
— No formal training program with lesson plans, etc.
• Vendors
— No briefing for escorted vendors

18 © 2015 Uptime Institute, LLC

Preventing these problems

© 2015 Uptime Institute, LLC

 Owners recommendations—Design
• Invest time and budget for pre-design, design, CX
and operations planning
— Document IT capacity plan
ƒ Forecast for now, middle, future scenarios
— Document facilities requirements
ƒ Redundancy, resiliency, technology preferences
ƒ Operations plan
— Executive briefing
ƒ How design will incur cost but deliver business value

20 © 2015 Uptime Institute, LLC

 Owners and owners representatives—
RFP and Contractors
• Build project team based upon design intent, technologies, and experience
• Structure contract for mutual success - Align owner and contractor objectives to
partner for success

• Sophistication of owner can drive team structure

— Owners representative role
— 3rd-party testing and verification
• Establish and manage a communication system
— Notify of any changes in facilities requirements or IT capacity planning
— Regularly review any proposed changes by the operation to normal configuration,
materials or controls

21 © 2015 Uptime Institute, LLC

 Operations Recommendations
• Change management is a critical life-cycle function of the
data center from design, through construction, CX, and
Operations
• Follow Uptime Operational Sustainability model to prevent the
common operations missteps:
— Inadequate staffing
— Ineffective or non-existing maintenance and training
programs
— Lacking processes and procedures
— Resulting in the majority or outages being caused by
‘human error’
22 © 2015 Uptime Institute, LLC
 It’s a Journey—Not a Destination
• Large enterprise build projects have tremendous risk and complexity
• Start with a clear project objective, partner with the team through completion
• Thorough Commissioning by independent 3rd party
• There are many quality colo and wholesale providers in the North America
market

• Operations is unlike other aspects of project development

- Not a one-time set up
- Regular assessments must be ongoing

• Certifications are a benchmark, not the destination

23 © 2015 Uptime Institute, LLC

 Questions?
Eric Maddison
Senior Consultant EMEA

emaddison@uptimeinstitute.com

© 2015 Uptime Institute, LLC