Data Centre Facility Design : Standard & Guideline

Session 1

• Importance of Data Centre

• Cause and Cost of Down-Time
• Topology Standards and Certification
• Tier-I / Rated-1 : Basic Component
• Tier-II / Rated-2 : Redundant Components
• Tier-III / Rated-3 : Concurrently Maintainable
• Tier-IV / Rated-4 : Fault Tolerant
Revolution Towards Industry 4.0
Critical Components of Data Centre

Power
Quality

Power
Security
Backup

24/7/365
Operation,
Access and
Support

Redundancy Cooling

Internet
Data Centre in a Business Process
 Operations of Data Centre
Data Centre Operations is a broad term that includes all processes and operations performed
within a data centre. Typically, data centre operations are distributed across several categories,
such as:

Infrastructure Operations
Installing, maintaining, monitoring, patching and updating server, storage & network resources

Security
Processes, tools and technologies that ensure physical and logical security in the data centre
premises

Power and Cooling

All processes that ensure enough power is supplied to the data centre facility and the cooling
system is operational

Management
Creation, enforcement and monitoring of policies and procedures within data centre processes
Cost of Down Time and It’s Calculation
According to Gartner, average cost of network downtime is around USD 5,600 per minute.
Considering average size of ISPs around the world. That is around USD 300,000 per Hour. For
any business, USD 300,000 per Hour is a lot on the line.

Beyond the monetary costs, IT downtime can wear on your business’s productivity levels.
Every time you get interrupted, it takes on average 23 minutes to get refocused on your prior
task.

Network failures and power outages aren’t the only culprits when it comes to downtime
either. Other top factors include:

 Outdated Software and Hardware

 Hurricanes and Floods
 Human Error

So how do you know where you stand when it comes to downtime costs? Here is a simple way
to calculate how your business could be affected:
Cost of Down Time and It’s Calculation
Cost of Downtime (/Hr) = Lost Revenue + Lost Productivity + Recovery Cost + Intangible Cost

Lost Revenue = Revenue/Hour x Downtime (Hour) x Percentage of Impacted Uptime (%)

Lost Productivity = [Employee Salary/Hr x Downtime (Hr)] x [#. of Employee x Utilization (%)]

Recovery Cost
These are the costs accrued while fixing the issue. They can include but are not limited to:
• Repair services ± Replacement parts
• Lost data recovery ± Other costs due to loss of data
These may not be as tangible at revenue and productivity costs, but they are equally as
important when deducing your real downtime costs.

Intangible Cost
These are the costs that can sting the most for the long-term. These occur when downtime
damages your reputation or your brand. These costs ultimately affect businesses that rely
heavily on uptime. Including intangible costs into the Total Downtime Cost Formula gives a
better understanding of the long-term consequences that can occur due to downtime.
Cost of Down Time : Ponemon Institute

o Average cost of data centre downtime across industries in 2018 was approximately USD
9,000 per Minute for an average of 2,500 m2 of White Space. (It was USD 5,600 in 2010.)

o Average reported incident length was 86 minutes in 2018, resulting in average cost per
incident of approximately USD 790,000 (In 2010 it was 97 minutes at USD 505,500)

o For a total data centre outage, which had an average recovery time of 119 minutes in 2018,
average costs were USD 1,901,500 (In 2010, it was 134 minutes at USD 680,700)

o For a partial data centre outage, which averaged 56 minutes in length in 2018, average
costs were USD 750,400 (In 2010, it was 59 minutes at approximately USD 258,000)

o The majority of survey respondents reported having experienced an unplanned data centre
outage in the past 24 months (91%) in 2018. This is a slight decrease from the 95 percent of
respondents in the 2010 study who reported unplanned outages.
 Example of Down Time : 2017
in May 2017, over 75,000 people had their three-day weekend plans stalled when British
Airways suffered a massive data centre failure. Along with the chaos of lost luggage,
broken trust and frustrations resulting from cancelled and delayed flights, the UK’s largest
airline also had to deal with a loss over $75 million. The outage was caused by a single
engineer who disconnected and reconnected the power supply in a disorganized fashion,
triggering a power surge that disrupted the operations of the entire BA infrastructure.

Even the data centre behemoths aren’t immune to outages– including Amazon Web
Services (AWS) which is considered to be the world’s biggest cloud provider, and home to
some of the biggest names on the Internet, like Slack, Quora, Netflix, and Airbnb. In 2017,
a mistyped command entered by an AWS engineer caused many sites to shut down for
several hours, prompting a loss of over $150 million.

A similar incident echoed a year later at one of AWS’ biggest rival. On September 2018, a
lightning strike caused Microsoft Azure to suffer an outage from voltage surge resulting in
damage to hardware, including network devices and storage units.
Example of Down Time : Google Cloud

In July, 2018 due to a bug in the new security feature added Google Front Ends (GFE)
architecture layer. The bug had not been identified earlier despite extensive testing
procedures in place, and was triggered only when the configuration changes were
introduced in the production environment. The affected services included the Google App
Engine, Stackdriver, Diagflow and Global Load Balancers. Customer including Spotify,
Discord, Pokemon Go app and Snapchat rely on these cloud networking services to reach
a global audience, thereby cascading the impact globally. The outage lasted for around 30
minutes and up to 87 percent of the customers experienced some form of errors on the
App Engine, HTTPS Load Balancer or the TCP/SSL Proxy Load Balancer solutions.

The affected customers were provided credits refund as per the Service Level Agreement
(SLA) as a common compensation by any cloud vendor. However, the true cost of data
centre downtime that averages around $750,000 as of 2015 according to a Ponemon
Institute research report far outweighed the offered compensation.
 Example of Down Time : O2

In terms of scale, the largest network outage impacted customers of O2 3G and 4G

mobile services in the UK. The outage starting early hours on the December 6, 2018
left 30 million users without the Internet connectivity capabilities. The outage lasted
the entire day and was caused due to failure on networking equipment operated by
Ericsson and served to several carriers globally. Considering the scale of the issue,
Ericsson readily worked to fix the issue and decommission the faulty software later
on. Detailed analysis showed that the root cause was tied to expired certificate
versions linked to customers including O2. Moreover, the Identical problem occurred
for same glitch in October, 2018. Although the issue lasted only 40 minutes.
 Example of Down Time : CenturyLink

The CenturyLink incident was the highlight network outage of 2018 as it left millions
of users without the ability to call 911, ATM withdrawals, access to sensitive patient
healthcare records, Verizon mobile data services and even lottery drawings. The
incident later led to an FCC investigation considering the “unacceptable” downtime
impacting emergency services such as 911 or ATM withdrawals. The outage lasted
two days and was caused due to an issue with a single network management card.
The device was found transmitting invalid data frame packets across the
infrastructure. Despite the multiple layers of redundancy in place, the issue cascaded
across its nationwide communication infrastructure. Once the infrastructure systems
crashed, CenturyLink had limited visibility into its network to troubleshoot the issue.
 Example of Down Time : 2019

Outages like these are not anomalies. The rate and severity of these incidents have
grown significantly over the last year (2018), according to Uptime Institutes’ Global
Data Centre Survey. What’s surprising is that 80% of the survey respondents believed
that their recent outage was preventable.

In March of 2019, the Nordic metals firm, Norsk Hydro, suffered a ransomware attack
called LockerGoga that shut down its global operations. This left their 35,000
employees around the world unable to progress with their work. At this point, they
are still working to calculate the financial impact of the attack, loss of wages,
productivity, and stock price drop.
UpTime Institute Industry Survey, 2018
UpTime Institute Industry Survey, 2018
UpTime Institute Industry Survey, 2018
UpTime Institute Industry Survey, 2018
Complexity of DC even without ICT
Risk Factors for Data Centre
Prominent Cause of Downtime/Failure
 Summary

Session Review Moral of the Story

DC is an integral part of Business DC plays a crucial role in High-Availability

Problems in DC could lead to business

DC has to be part of, and accommodate,
incurring substantial loss to the point of
the fast pace of Change in Demand
total closure

Fact is that most of the DC today are not

DC is a complex and dynamic space
ready for Tomorrow’s Demand

Most Down-Time is caused by Power, Most, if not all, DC have experienced

Cooling, EMF and/or Human Error changes making them potential risk
[only considering Facility Down-Time] factors for achieving High-Availability
22
 Standards and Certification
Standard → UpTime EPI based on BICSI based on SS-507 EN-50600
↓ Guideline [USA] TIA-942 [USA] TIA-942 [USA] [Singapore] [Europe]
Conformity Tier : I - IV Rated : 1 - 4 Class : 0 - 4 Pass / Fail Class : 1 - 4
Availability of
Yes Yes (Paid) Yes Yes Yes
Standard
Certification Available Available Not Available Available Available
Electrical Electrical Electrical Electrical
Tier Standard
Mechanical Mechanical Mechanical Mechanical
Electrical
Distribution Distribution Distribution Distribution
Mechanical
Scope of Architectural Architectural Architectural Architectural
Distribution
Topology Telecom Telecom Telecom Telecom
Site Location Site Location Site Location Site Location
OS Standard
Safety-Security Safety-Security Safety-Security Safety-Security
Other Element
Efficiency Efficiency
Incorporation Commercial Non-Profit Non-Profit Non-Profit Non-Profit
Accreditation No ANSI ANSI Spring EN-CENELEC
Training Event Yes Yes Yes No No
Auditor UpTime Only Multiple ORG N/A Multiple ORG N/A
Topology Standard : Definition
Tier Topology : Basic, RC, CM
Topology Standard : N, N+1, N+2
Topology Implementation : N+1
Redundancy : Component vs Path
Tier Topology : Fault Tolerant
Topology Deployment : Fault Tolerant
 Topology Standard : UpTime and EPI
Tier I Tier II Tier III Tier IV
Active Capacity Components N
N N+1 N+1
to Support the IT Load After Any Failure
Distribution Paths 1 1 1 Active and 1 Alternate 2 Simultaneously Active
Concurrently Maintainable No No Yes Yes
Fault Tolerance No No No Yes
Compartmentalization No No No Yes
Continuous Cooling No No No Yes

Rated 1 Rated 2 Rated 3 Rated 4

Active Capacity Components
N N+1 N+1 N+N
to Support the IT Load
Distribution Paths 1 1 1 Active and 1 Alternate 2 Simultaneously Active
Concurrently Maintainable No No Yes Yes
Fault Tolerance No No No Yes
Compartmentalization No No Yes Yes
Continuous Cooling No No No No
Simplest SLD : Basic Component
Simplest SLD : Redundant Component
Simplest SLD : Concurrently Maintainable
Real Life SLD : Tier-III Compliant
Simplest SLD : Fault Tolerant
 Session 2
• Environment Class Rating
• Engine Generator Rating
• Hybrid Topology : Best Practice
• Hybrid Topology : Definition
• Hybrid Topology : Design Requirement
• Objective and Summary of Certification
• Cooling Technology for Network Node
• Plenum Cooling Technology
ASHRAE, 2011 : Environment Class
ASHRAE, 2011 : Data Centre
Environmental Class Rating

UpTime does not talk about Environmental Class Rating

 Engine Generator Rating

EPI requires Engine Generator to be Prime with Nameplate Capacity

 Hybrid Topology : Best Practice
Level 1 Level 2 Level 3 Level 4
Active Capacity Components
N N+1 N+1 N+N
to Support the IT Load
Distribution Paths 1 1 2 (Both Active) 2 (Both Active)
Concurrently Maintainable No No Yes Yes
Fault Tolerance No No No Yes
Compartmentalization No No Yes Yes
No [Average < 5 KW]
Continuous Cooling No No Yes
Yes [Average > 5 KW]
Site Selection EPI EPI EPI EPI
Civil / Structural EPI EPI EPI EPI
Architectural EPI EPI EPI EPI
MMR and Structured Cabling EPI, IBP EPI, IBP EPI, IBP EPI, IBP
Safety, Security, Fire Code AHJ, EPI AHJ, EPI AHJ, EPI AHJ, EPI
Efficiency [PUE, WUE, CUE, DCIE] TGG TGG TGG TGG
 Hybrid Topology Definition : Level 3
Compliance
Validation and Audit
Standard

‘Concurrent Maintainability’ is the philosophy behind the Rated-3 and Tier-III

conformity. It requires to ensure that every capacity component as well as their
distribution path can be removed / replaced / serviced on a planned basis
without disrupting the ICT capabilities to the end-user.

It applies to all active and passive components of MEP infrastructure. However,

Rated-3
Architecture-Civil, Fire Suppression and Safety-Security provisions are out of this
and Tier-III
scope. Manual fail-over switching of electrical-power is allowed.
Facility
Furthermore, it requires that each distribution path for power, cooling, ICT to be
physically separated. Specifically transformer, generator, UPS, battery, chiller
plant, carrier room / meet-me room and rack-space should remain 1 (one) hour
fire-separated from each other. Additionally, no sharing of PDU, fire suppression
and cooling is allowed as well.
 Hybrid Topology Definition : Level 4
Compliance
Validation and Audit
Standard

‘Fault Tolerant’ is the philosophy behind the Rated-4 and Tier-IV conformity. It
requires to ensure that every capacity component in either of their
distribution path can run on the full-load operation of the facility. Hence,
capacity component as well as distribution path can tolerate a fault anywhere
in the system while the facility is having planned down-time / maintenance
without disrupting ICT capabilities to end-user.

Rated-4 and It applies to all active and passive components of MEP infrastructure. However,
Tier-IV Architecture-Civil, Fire Suppression and Safety-Security provisions are out of
Design this scope. Software tools for remote operation is required.

Furthermore, it requires that each distribution path for power, cooling, ICT to
be physically separated. Specifically transformer, generator, UPS, battery,
chiller plant, carrier room / meet-me room and rack-space should remain 2
(two) hour fire-separated from each other. Additionally, no sharing of PDU, fire
suppression and cooling is allowed along with manual fail-over switching of
electrical-power.
 Hybrid Topology : Requirement
# Site Selection Requirement
Ground Floor should be high-enough to sustain any flash flood based on 50 (fifty) years of
1
flood history
2 Distance from Air-Port should be 8 Km / 5 miles
3 Distance from Rail-Station should be 0.8 Km / 0.5 miles
4 Within 3,050 m / 10,000 feet from the sea-level
5 Capability to Handle Seismic Activity based on ‘Zone’ Requirement
Away from Chemical Plant, Power Generation Plant and Establishment which could be
6
categorized as ‘Potential Target of Attack’

# Metallurgical, Structural, Civil and Architectural Requirement

1 No use of Asbestos, Lead and Poly-Chlorinated Biphenyl
2 Permanent Shelter with Class-A roof (fully adhered roof)
3 Separate Parking for Employees and Visitors with Zoning
4 IBC-2006 (+ updates), NFAP-2001 (+ updates) to be followed along with Code of AHJ
5 Legal Hierarchy of Compliance : Law / Code of AHJ, DC Standards, International Standards
45
 Hybrid Topology : Requirement
# Metallurgical, Structural, Civil and Architectural Requirement
6 Rated-3 requires IIA / IIIA / VA and Rated-4 requires IA / IB level of construction
7 Vapour Barrier with plastic-sheet or latex-paint
8 Roof Slope for Rated-3 is 1:48 and for Rated-4 is 1:24
9 Floor Loading for Rated-3 and Rated-4 is 12 KN/m²
10 Suspended Ceiling is not recommended
11 All the MEP and ICT equipment should be based on ‘Seismic Zone’ Requirement
12 No Exterior Windows in Data Hall / Computer Room
Access Control, Surveillance System and Fire Suppression System shall not be shared in
13
case of the fire-separated zone / room
Monitoring should be continuous (recording should be activity based) along with
14
minimum of 20 fps. Hence, PTZ camera is not acceptable
15 Dual-Factor authentication based Access Control
16 Single-Person Double Door Interlock based Man-Trap at the Entrance of Data Centre
17 Bullet Proof (Level-3 of UL-752) Window at the Reception / Guard Room (1st Entrance)
46
18 Doors should swing-away while getting out of any room (if Local Fire Code allows it)
 Hybrid Topology : Requirement
# Metallurgical, Structural, Civil and Architectural Requirement
Rated-3 requires 24/7 presence of security guards and Rated-4 requires 24/7 presence of
19
both armed and security guards with walk-in escort and inspection patrol
Emergency Exit Sign, Exit Path Direction, etc. requires minimum 1 Lux light at all time.
Unoccupied Rack-Space requires light to keep Surveillance System running smooth.
Occupied Rack-Space requires 200 Lux (measured at 1m above the ground and 1m away
20
from the rack) at the sides of the racks and 500 Lux at the front and back of the racks.
However, use of night-vision camera and motion sensor will allow to go completely dark
rack-space (Emergency Exit Sign, Exit Path Direction shall remain switched on at all time)
Reception / Entry-Lobby, UPS, Battery and Gen-Set rooms should have 2 (two) hour fire-
separation for Rated-3 and 4 (four) hour fire-separation in case of Rated-4 facilities. For
21
the rest of the walls, floors and ceilings of the fire separated zone / room it should be
respectively 1 (one) and 2 (two) hours
Exclusive ramp / bay for Loading-Unloading and Shipping-Receiving should be followed by
22
Warehouse and Staging area
Two entry-road and loading-bay along with One (sole) reception and exit-road for vehicle
23
and human movement
24 Access Path and Exit Corridor Philosophy ‘What Come In Should Be Able to Go Out Again’
47
 Hybrid Topology : Requirement
# Mechanical, Electrical, Plumbing (MEP) Requirement
1 Rated-4 requires a Remote Operations Centre, Remote Switching of Light and PA system
DCIM (WLDS, EMS, BMS and Automatic Power Fail-Over Switch) or, IDC-IMS (DCIM and
2
NMS) shall be used to mitigate the risk for Rated-4 Facility
3 EMF should be isolated / filtered and it should be measured from Rack PDU level
4 UPS and Battery rooms should have maintenance-aisle of minimum 1.2 meter
5 Rack to Rack distance both in hot and cold aisle should be minimum of 1.2 meter
DRUPS, Generator, Chiller, Cooling Tower, Valve, Fire Pump, Diesel Pump and Chilled
6
Water Pumps should have maintenance-aisle of more than the width of equipment
Battery room need to be separated from UPS room, Switchgear room and Generator
7 Plant. It’s also needs to have sufficient natural-air ventilation and Shatter-Proof glass
window in the door
Rated-3 allows the Generator / DRUPS to be placed in the Data Hall building with
8 compliance of fire separation. Whereas, Rated-4 requires the Generator / DRUPS to be in
separate building / weather proof enclosure with compliance of fire-separation
Diesel Reservoir and Water Reservoir need to carry at least 12 (twelve) hours of full-load
9
operation along with redundancy in storage tank, pump and plumbing. 48
 Hybrid Topology : Requirement
# Mechanical, Electrical, Plumbing (MEP) Requirement
Generator / DRUPS Rooms and Diesel Reservoirs should be 9 m / 30 feet away from public
10
area in case of Rated-3 and the distance should be 19 m / 60 feet in case of Rated-4
Data Halls should not have any external window, internals are allowed if they meet the
11
fire-ratings and security
Rated-3 requires everyone to face 3 (three) access control till the Rack and Rated-4
12
requires 4 (four). However, recommendation is to have 1 (one) extra stage for both cases
13 Minimum 46 cm / 18 inch vertical clearance under fire suppression nozzles till 1st obstacle
14 Underground Up-Stream Utility Power Feeder is preferred over Overhead (optional)
Rated-3 requires Single Sub-Station (with Dedicated Distribution Feed for the Facility from
Sub-Station) with N+1 Down-Stream Power Feeder. And, Rated-4 requires Dual Sub-
15
Station (with Dual Dedicated Distribution Feed for the Facility from Sub-Station) with 2N
Down-Stream Power Feeder
Both up-stream and down-stream power-feeders will remain separated by 20 meter till
16
they are exclusively distributed inside Data Halls
Both up-stream and down-stream optical-fibres will remain separated by 20 meter till
17
they are exclusively distributed inside Data Halls 49
 Hybrid Topology : Requirement
# Mechanical, Electrical, Plumbing (MEP) Requirement
18 UPS should be backed by Prime Generators (at least) not Standby Generators
Cogeneration Plant instead of 2nd Utility should use Continuous Generator neither Prime
19
nor Standby
20 HT Switchgear Panel should have Surge Protection Device and Draw-Out Circuit Breakers
Generator with Battery Bank and UPS / DRUPS should be of N+1 configuration (where, N
= 1 - 9. Hence, +1 redundant-component for every 9 needed-component) for Rated-3.
21
Whereas, it is 2N configuration (N has no limit) for Rated-4. Furthermore, both Rated-3
and Rated-4 requires Dual-Bus system of Distribution
Chiller, Cooling Tower, AHU, In-Row AHU, Chilled Water Pump, etc. should be of N+1
configuration (where, N = 1 - 5. Hence, +1 redundant-component for every 5 needed-
22
component) for Rated-3. Whereas, it is 2N configuration (N has no limit) for Rated-4.
Furthermore, both Rated-3 and Rated-4 requires Dual Piping system of Distribution
Rated-3 and Rated-4 both requires a dedicated feeder in to the Automatic Bypass of the
23
UPS and a dedicated Maintenance Bypass Feeder serving the UPS output PDU
If Static UPS (battery-bank) is used Rated-3 requires minimum 10 (ten) minutes and
24
Rated-4 requires minimum 15 (fifteen) minutes of full-load operation 50
 Hybrid Topology : Requirement
# Mechanical, Electrical, Plumbing (MEP) Requirement
If DRUPS (fly-wheel) is used Rated-3 requires minimum 6 (six) seconds and Rated-4
25
requires minimum 9 (nine) seconds of full-load operation
ICT equipment, Safety-Security, Automation Software, Emergency Light and Signs, Chilled
Water Pump and AHU should be under the UPS Power Supply. However, ICT and
26 Mechanical Loads are recommended to serve with 2 (two) separate Distribution Network
of N+1 configuration for Rated-3 facility. Whereas, Rated-4 requires the same provisioning
of 2 (two) separate Distribution Network of 2N configuration
Chiller, Office Equipment, Lighting Load, Diesel and Water Pump, etc. should be under the
Gen-Set Power Supply with dedicated Distribution Network of N+1 configuration for
27
Rated-3 facility. Whereas, Rated-4 requires the same provisioning of Distribution Network
of 2N configuration
Power Strips / Metered PDU / Tap-Off Box should also carry K-Rated Isolation Transformer
28
or IGBT Harmonics Filter to eliminate the Common Mode Noise
29 Grounding and Signal Reference Grid as per IEEE Standard
HT and LT Panel along with Generator / DRUPS and each level of electrical distribution is
30 recommended to have Surge Protection Device. Whereas, the requirement is to have
Surge Protection in the MDB of Low Voltage Side 51
 Hybrid Topology : Requirement
# Mechanical, Electrical, Plumbing (MEP) Requirement
Generator-UPS / DRUPS, SPD, HT and LT Switchgear Panel, Motor Control, Chiller Plant,
31 AHU, ATS, STS, Sensors, etc. are recommended to be monitored by CPM system (which
needs to have Multiple Level, Multi-Channel Notification)
32 Rental Provisioning of Load Bank is required (for Testing, Audit and Initial Low-Load Stage)
33 Current Rating : [Phase 1 = Phase 2 = Phase 3 = Ground = 100%] and [Neutral = 200%]
34 Isolation Transformer [K-13 or Higher] and/or IGBT (in PDU) to be used for ICT Equipment
Common Mode Noise (CMN) is acceptable up to 1% of Phase to Neutral Voltage.
35
Maximum Acceptable CMN is 3 Volt and Preferred CMN is less than 1 Volt
36 Form Factor of Panel / Distribution Boards are to be of 2B for Rated-3 and 3B for Rated-4
37 Data Hall Design should be designed as MICE-1 (M1I1C1E1)
38 Environmental Design should be designed as M1I2C1E3
39 HVAC System Design should be designed as per ASHRAE, 2011 (including Updates)
40 Fire Suppression Design should be as per NFPA 2001 (including Updates) and Code of AHJ
41 Telecom and Network Design as per TIA Standards
42 Data Halls should have Positive Air Pressure 52
Objective of Certification
Summary of Tier Certification
Types of Heat

55
Cooling Technology for Network Node

Free Cooling Chemical Coolant (DX) Water Coolant

Technology Technology Technology

Comfort AC
Natural Free Air Cooled Water
VRF
Cooling (+DX) Chiller
Precision AC

Kyoto Wheel Air Cooled DX Water Cooled

Strainer Cycle (+Water) Chiller Water Chiller

Not Applicable Applicable Applicable

[Weather Dependency] [Small & Medium] [Medium & Large]
56
Free Cooling Technology
Chemical Coolant Technology : Plenum
Chemical Coolant Technology : In Row
Water Coolant Technology
 Trend in ICT : Power and Cooling

Statistics of Yearly Bangladesh

Average Per Rack Power Density
Statistics of Yearly World Average
Per Rack Power Density [* IBM]

• Year 1995 1.1 KW • Year 1995 No Data

• Year 2000 2.2 KW • Year 2000 1.1 KW
• Year 2005 3.5 KW • Year 2005 1.6 KW
• Year 2010 4.4 KW • Year 2010 2.2 KW
• Year 2015 5.5 KW • Year 2015 2.8 KW
• Year 2020 7.0 KW • Year 2020 3.5 KW
• Present Maximum Power • Present Maximum Power
Density / Rack 30 KW Density / Rack 7.0 KW
Importance of Structured Cabling
Relation : Power, Cooling and Heat

Power | Heat

1 KW of Power Consumption = 1 KW of Heat Generation

Heat | Cooling

1 KW of Heat Generation = 1 KW of Cooling Requirement

Cooling | Power

1 KW of Cooling Requirement ≠ 1 KW of Power Consumption

Cooling Technology Selection

VRF AC
Comfort AC
1 – 20 TR
Immersion
Precision AC
30 – 130 KW
20 – 100 TR
Power / Rack

Cooling
Technology

DX (± DF)
WCWC (±DF)
Chiller
400+ TR
100 – 200 TR

ACWC (± DF)
200 – 400 TR
Plenum Cooling Technology
Plenum Cooling Technology
Plenum Cooling Technology
Plenum Cooling Technology
Plenum Cooling Technology
Plenum Cooling Technology
Plenum Cooling Technology
 Session 3

• Static UPS Technology

• Battery Technology
• Rotary UPS Technology
• Comparison Between UPS Technology
• Power System Configuration (Gen-Set, UPS)
Static UPS System : Type and Mode
Static UPS : Offline / Standby (VFD)
Static UPS : Line Interactive (VI)
Static UPS : Online / Double Conversion (VFI)
Static UPS System : Pros and Cons
Static UPS System : Summary
Battery Technology : Flooded Cell
Battery Technology : SLA / VRLA
Battery Technology : Ni-Cd / Lithium Ion
 Rotary UPS System : Type and Mode

Rotary UPS Operational Mode

1. Utility Mode
2. Change Over to Diesel Mode
3. Diesel Mode
4. Back To Utility Mode
Rotary UPS System : Operational Basic
Rotary UPS System : Operational Basic
Rotary UPS System : Operational Basic
Rotary UPS System : Pros and Cons
 Comparison : SUPS versus DRUPS
DRUPS Diesel Rotary UPS + LT-Panel

SUPS Static UPS + Battery Bank + Battery Room PAC + PFI + Many Distribution Panel + AVR +
Generator + UPS and Gen-Set Synchronizers + Phase Plotter

Description DRUPS [1.5 MW] SUPS [1.5 MW]

Capital Expenditure [CapEx] 1.0 Million USD 0.75 Million USD
Battery Backup Not Required Required (5 min)
Space Consumption 100% 182%
Life Time 25 Years 10 Years
Operational Expenditure [OpEx] 1.84 Million USD 5.46 Million USD
Pay Back [CapEx Difference] 9 Months 0 Months
Power Efficiency [AVR to BBT] 97% 92%
Continuous Cooling Possible Not Possible
Power Level 415 Volt | 11 KV | 33 KV 415 Volt
NTT, KDDI, Fujitsu, AWS, Global Equinix, TM, CT, MS, AWS, Tata,
Signature Users
Switch, GE, Tele House, Google Digital Realty, Facebook, Google
Market Share [as Clean & QPS] 18% 82%
Total Power > 3 MW Total Power < 3 MW
Thumb Rule of Usage OpEx is 1st Priority CapEx is 1st Priority
Continuous Cooling Backup Time > 30 s
Power System Configuration
 Power System Configuration

Parallel Configuration Master-Slave Configuration

Power System Configuration

Cross-Link Configuration
Power System Configuration

Isolated Redundant Configuration

Isolated Parallel Configuration (IP Bus)

Power System Configuration

Distributed Redundant Configuration

 Power System Configuration

Power System Configuration Selection [N+1 Setup for Level-3 Certification]

1. Static UPS Distributed Redundant Configuration
2. Rotary UPS (LV) All Configuration remains Applicable
3. Rotary UPS (MV) All Configuration, But, Isolated Parallel Configuration (IP Bus)

Tier-IV Certification with N+1 Setup [Assuring ‘N’ after any Failure]
1. Static UPS Not Applicable
2. Rotary UPS (Only LV) Only Applicable with Isolated Parallel Configuration (IP Bus)
 Session 4

• Raised Floor, Floor Loading and SRG

• TIA-942 Cabling Standard
• Cable and Pipe Distribution
• Seismic Isolation for Rack and Facility
• Data Centre Efficiency : The Green Grid
• Safety, Security, Monitoring in Data Centre
• Fire Suppression System
• Technology Selection : Option & Thumb Rule
• Quiz : Topology and Design Misconception
• Question, Answer, Feedback, Advice
 Raised Floor and Type of Loading

Raised Floor Selection

1. Die Formed Welded Steel Construction [Level-1 and Level-2]
2. Die Formed Welded Steel Shell with Cement-Filled Core [Level-3 and Level-4]
3. Galvanized Floor [Beware of Zinc Whiskers | Not Recommended]
4. Wood Filled Core [Beware of Rolling Load, Life | Not Recommended]
5. High Pressure Laminate [Recommended for Rack-Space]
6. Rubber Laminate [Recommended for Ramp, Access Floor, NOC, etc.]
Load Bearing Capacity : Slab and Tiles

Do Not Forget to Add the Weight of the Raised Floor to Building Floor Load
Signal Reference Grid and Grounding
Raised Floor Guidelines
1. Height : 300 mm – 1 m
2. Ramp Slope = 1 : 12
3. Ramp Width : 600 mm
4. Aisle Width : 600 mm
5. Wheel Chair Road : 1 m
6. Rand Rail beside Ramp
7. No Plumbing (Optional)
Bonding-Earthling Guidelines
1. Individual Device Bond
2. Serial Bonding : NA
3. IEEE-1100 to Follow
4. Ground < 1 Ohm [9 Hole]
5. Code of AHJ to Follow
 TIA-942 Cabling Standard

ER : Carrier MDA : MMR HDA : DH-MMR ZDA : End of Row HDA : Top of Rack EDA : Rack
TIA-942 Cabling Standard
TIA-942 Cabling Standard
 TIA-942 Cabling Standard

Peering Security Aggregation Data Centre

 TIA-942 Cabling Standard

TIA-942 Cabling Standard Requirement

Structured Cabling Applicable for Enterprise DC
On-Demand Cabling Applicable for Internet DC / Colocation DC
Copper Media Cat 7, 7A or Higher
Fibre Media (Single Mode) OS2 or Higher
Fibre Media (Multi Mode) OM4 or Higher
Fibre Termination (Patch Panel) MPO
Building to Building Connectivity (Fixed Site) Hard Wired
Building to Building Connectivity (Fixed Site) Canopy (Point to Point / Access Point)
Building to Building Connectivity (Temporary) Free Space Optics (Requires LOS)
Cable and Pipe Distribution

- Solid Tray as Drain to Over-Head Water Pipe

- RCC Duct for Underground Cable Tunnel

Data Centre Efficiency : The Green Grid
Data Centre Efficiency Scale
PUE Calculation : Thumb Rule
Cooling Technology PUE
Natural / Free Cooling 1.3
Comfort AC 1.9
VRF System 1.85
Precision AC 1.8
DX Chiller (Scroll Compressor) 1.75
Air Cooled Chiller (Screw Compressor) 1.6
Water Cooled Chiller (Magnetic Compressor) 1.45
Water Cooled Chiller (Screw Compressor) 1.6
Water Cooled Chiller (Centrifugal Compressor) 1.7

Measurement Green DC Inefficient DC

PDU Level PUE < 2 | DCIE > 50% PUE > 2 | DCIE < 50%
Physical Security & Risk Management

Perimeter
Security,
Reception,
Patrolling
Rodent Baggage
Repellent Scan, Metal
and Pest Detector,
Control Integrated Man Trap
Security
Management
with 24/7/365
Fire Operations
Detection Centre
Access
and
Control
Suppression
System
Surveillance
System
Access Control & Surveillance System

Layer 1
Perimeter
Layer 6
Layer 2
White Space
Clear Zone
/ Cage /
and Parking
Containment
Layer 7
Rack Doors
[Front & Back]
Layer 5 Layer 3
Hall Way / Façade &
Gray Space Reception
Layer 4
Turnstile /
Man Trap
Layer 4 : Tailgating and Piggybacking
Data Centre Infrastructure Management
Fire Triangle
Fire Suppressant Gas : Modus Operandi

There are four means used by the agents to extinguish a fire. They act on the
"fire tetrahedron":

• Reduction or isolation of fuel. No agents currently use this as the primary

means of fire suppression.

• Reduction of heat. Representative agents: Clean agent FS 49 C2 (NAF S 227,

MH227, FM-200), Novec 1230, pentafluoroethane (NAF S125, ECARO-25).

• Reduction or isolation of oxygen: Representative agents: Argonite / IG-55

(ProInert), CO2 carbon dioxide, IG-541 Inergen, and IG-100 (NN100).

• Inhibiting the chain reaction of the above components. Representative

agents: FE-13, 1,1,1,2,3,3,3-Heptafluoropropane, FE-25, haloalkanes,
bromotrifluoromethane, trifluoroiodomethane, NAF P-IV, NAF S-III, NAF S
125, NAF S 227, and Triodide (Trifluoroiodomethane).
Fire Suppression System
Fire Class
Fire Class
Fire Suppression Primary System : Gas
Agent → Halon 1301 FM 200 Inergen
CO2 NOVEC 1230
↓ Features [FE 13 / HFC 23] [NAF S 125] [Argonite]
Jan 1, 1994
Production In Production Jan 1, 2022 In Production In Production
[In Production]
Ozone (65 Year) 33 Years
Environment Natural Gas 5 Days Natural Gas
[243 Years] [29 Years]
- 29 C to 54 C 0 C to 65 C
Temperature - 15 C to 51.5 C - 15 C to 100 C - 17 C to 54 C
[- 20 C to 50 C] [0 C to 54 C]
Pressure 60 Bar 35 Bar 25 Bar 40 Bar 150 Bar

Flooding Height 3.5 m [8 m] 3.0 m 3.5 m 4.0 m 4.0 m

Flooding Time 10 Sec / Less 10 Sec / More 10 Sec / Less 10 Sec / Less 60 Sec / Less
Fire Zone Not Possible Not Possible Not Possible Not Possible Possible
Working Oxygen Reduce Heat Absorb Oxygen Reduce
Heat Absorb Heat Absorb
Principle (15%) [Heat + Oxygen] (12.5%)
9% (7.5%) 5% (35%) 9% (7.5%) 43% (40%)
NOAEL (Design) 10% (5%)
[30% (15%)] 9% (B), 34% (D) [7.5% (5%)] [43% (40%)]
Usage in DC Banned [Partial] Mostly Banned Partial Ban In Use In Use
Look / Smell Clean White Mist Clean [Lemon] Clean Clean
Fire Suppression System : Selection

Large (> 1000 m²) Small (< 1000 m2)

1st (Mandatory) 1st (Mandatory)

Inergen / Argonite ; or, NOVEC 1230 (Green); or,
FE-13 (for High Void, Risky)

2nd (Optional) 2nd (Optional)

Pre-Action System Water Mist / Pre-Action
(Corrosion is a Problem) System (Corrosion Check)
Technology Selection and Options

Electrical and Power System Mechanical and HVAC System

Static UPS (Modular / Stand Alone) with

Water Cooled Water Chiller
DG, AVR, PFI, Battery Bank, Synchronize

Dynamic Rotary UPS Air Cooled Water Chiller

Flywheel UPS backed by Static UPS Precision Air Condition

Technology Selection and Options

Access Control System Fire Suppression System

Plum Vein + PIN + Access ID NOVEC 1230 + VESDA

Finger Vein + PIN + Access ID INERGEN + VESDA

Irish / Finger Print + PIN + Access ID NAF S-125 / Water Mist + VESDA
Technology Selection and Options

Power Distribution System Cold Air and Hot Air Separation

Continuous Bus Way + Cast Resin BBT Slab Floor, Chimney Return

Any Slab, Smart Aisle (Rack Sensing)

Continuous Bus Way + Sandwich BBT
Containment, Room Return
Any (Raised / Slab) Slab, Aisle (Hot /
Cast Resin / Sandwich BBT (both use)
Cold) Containment, Room Return
Transformation Towards Efficiency

Public Cloud Public POP

instead of instead of Infrastructure Sharing
Enterprise Enterprise

Public POP Public Cloud

Node Accumulation
to IDC to Cloud DC

UPS and
Cooling
Electrical Improvement in PUE
Efficiency
Efficiency
Quiz : Tier Topology Misconception

Raised Floor is Without Utility

Mandatory for Power Supply We
DC Certification Can Acquire Tier
Certification

YES / NO YES / NO

Tier-III IDC with UPS Setup of N+1

Average Per Rack with Isolated
Power Density is Parallel Bus
more than 4 KW, System, Can We
What Auditor will Acquire Tier-IV
Ask in TCCF from UTI in TCDD
Quiz : Design Misconception

What is the Right Without Safety,

Sequence: Security & DCIM -
Rack Door-Lock We Can Acquire
Data Hall Access Tier Certification
Man Trap [2 Factor]
Data Hall Turnstile YES / NO

Rated-3 requires TIA-942, USA

Dual Utility Awards DC
Power Source Certification

YES / NO YES / NO
Question, Comment, Feedback, Advice