ayn suyeuydy, i oerereeSnEnEREE EEE nREEEEINEnEEnneeeeeemeeeeeeeeeeeeeneneeeeeeee ES wosaynyasuIsWndn//:caaY @leubiseq Jel) peylpasooyCCEOCCCECOCECCECECOCECECCECCCCEOCCECCECOCCECCCCECCCC CEECopyright ©2009-2016 by Uptime Institute, LLC 20 West 37th Street - 6th Floor New York, NY 10018 Todos los derechos reservados. Las Publicaciones de Uptime institute (El Instituto) estan protegidas por leyes intemacionales de derechos de autor. El Instituto requiere de solicitudes por escrito en cada ocasién que la propiedad intelectual del Instituto 0 porciones de la propiedad intelectual del Instituto sean reproducidas o usadas. Los derechos de autor del Instituto se extienden a todos los medios de comunicacién: papel, electrénicos y contenido de video; ¢ incluye el uso en otras publicaciones, distribucién interna de la empresa, sitios web de la compaitia, materiales de marketing, y flletos para seminarios y cursos. Para obtener mds informacion, visite viww.uptimeinstitute.org/resources para descargar un Formulario de Solicitud de Permisos de Reimpresién de los Derechos de autor. Derechos de autor. IP Street, 6" Floor NY 10018 16. 705.4149 ‘upimeinsttvie comUptime Institute accredited tier designer™ Table of Contents Tier Standards and Technical Papers. Course Schedule... Session 1 Notes Session 2 Notes..... Session 3 Notes... Session 4 Notes. Session 5 Notes... Session 6 Notes... Session 7 Notes .. Session 8 Notes.. Abbreviations: Uptime Institute Contacts ..Co ¢ CCCE ccc ¢ ccc?yy DOD 2) xo] = Gj xo] i= © ~ ip)COOCOCCOCOECECECOECEC EEC ECEECOCECECOCOCOECCCEOCOCC CC CEE0 Institute Ee Mer Tamed CM ei ong) Tier Standard: TopologyUptime Institute’ Abstract ‘The Uptime Institute Tier Standard: Topology is an objective basis for comparing the functionality, capacity ‘and expected availabilty (or performance) of @ particular site Infiasirucure design topology against other sites, ‘or for comparing a group of sites. This Standard describes criteria to differentiate four classifications af site infrastructure topology based on inereasing levels of redundant capacity components andl distribution paths. “This Standard focuses on the definitions ofthe four Tiers and the performance confirmation tests for determining compliance to the defintions. The Commentary, in a separate section, provides practical examples of site Infrastructure system designs and configurations that full the Tier definitions as a means to clarity the Tier classification criteria. Keywords ambient temperatures, autonomous response, avallabiliy, classification, Compartmentalization, Concurcent Maintenance, Goneurrently Maintsinable, Continuous Cooling, data conter, dry bulb, dual power, Fault Tolerance Fault Tolerant, functionality, infrastructure, metrics, Operational Sustainabilly, performance, redundant, reliability, ‘Tier, Ter level, Tiers, topology. wet bulb Copyrights “This document is copyrighted by the Uptime Institute, LLC. Uptime Institute—in making this document avaliable ‘as a reference to governmental agencies, pubic institutions, and privata users—does not waive any rights in ‘copyright to this document. Uptime Institute's Publications are protected by intamational copyright law. Uptime Institute requires written requests at each and every occasion that Uptime Institute's inallectual property or portions of the intellectual property are reproduced or used. Uptime Insitute copyright extends to all mecla—paper, electronic, and video content—and includes use in other publications, internal company distibution, company Web sites and marketing ‘materials, and handouts for seminars and courses. For more information. please visit www.uptimeinstitte.com/oublications to dawnload a Copyright Reprint Permission Request FormA Gjolatertdeetsiataetc Introduction ‘This introduction is not part of Uptime Insitute Data Center Site infrastructure Ter Standard: Topology. It provides the reader wih context for the application of the Standard ‘This Institute Data Conter Site Infrastructure Ter Standard: Topology isa rastatement ofthe content previously published as Uptime Insttute publication Tier Classificalions Define Site Infrastructure Performance. Selected content of this publication has been reecited into an ANSI Standards Model format. Future updates or changes to Uptime Institute Tir Standard: Topology shall be accomplished through a review and recommendation process consistent with other recognized Standards bodies The Tier Classifications were created to consistently describe the site-level infrastructure required to sustain data Center operations, not the characteristics of individual systems or subsystems. Data centers are dependent ‘upon the successful and integrated operation of electrical, mechanical, and building systems. Every ‘subsystem and system must be consistently deployed withthe same site uptime objective to sais the istinctive Tier requirements. The most ertical decision-making perspective owners and designers mist consider wnen making inevitable tradeotts, is wat effect dows the decision have on the life-cycle-Intagratad operation of the Information Technology (IT) environment in the computer room. ‘Simply put, the Tier topolagy rating for an entire sie is constrained by the rating af the weakest subsystem that ‘il impact ste operation. For example, a site with a rebust Tier IV UPS configuration combined with Tier I chilled water system yields a Ter I site rating This very stringent definition is driven by senior executives whe have approved mult-milion dolar investments for an objective report of actual site capabilties. Any exceptions and exclusions footnoted in the approval documents Wl Be quicky lost and forgotten. Ita site has been advertised within an organization as being Fault Tolerant (Tier 1¥), i willbe inconsistent to have to plan a site shutdown at any time inthe future—regardless of any “ine print” exclusions that dligamtly identiied the risk, For this reason, there are no pattial or tractional Tier ratings. A site's Tier rating 's not the average of the ratings forthe critical site infrastructure subsystems. The site's Ter ‘ating isthe lowest of the individual subsystom ratings ‘Similarly, the Ther rating cannot be claimed by using caleviated mean time between falutes (MTBF) component statistical reliably to generate a prodictive availabilty and then using that number to mateh the fmpiical avalibility results with those of sites representing the difarontTiorslacefioationo, Statiotealy valid component values are not available, partly because product life cycles are gatting shorter and no independent industry-wide database axisis to collec failure data. Finally this Standard focuses on the topology and performance of an individual site. High levels of end-user availabilty may be atainod through the integration of complex IT architectures and network configurations that ‘ake advantage of synchronous applications running on multiple sites, However, this Standard is independent of the IT systems operating within the siteUptime Institute’ Additional Factors and Exposures Upiime Institute Tier Standard: Topology and Tier Standard: Operational Sustainabiliy establish a consistent ‘set of perlormance criteria that can be satislied, and agjudicated, worldwide. For the data center design, implementation, and sustained operation to be suecesstl, addtional factors and exposures must also be ‘considered by the owner and project team. Many ofthese will be dictated by the site location as well as local, national, of regional considerations andlor regulations. For example, building codes and Authorities Having Jaisdiction {AHUs); seismic, extreme weather (high winds, formado); loading: adjacent property uses; union fr ather organized labor force; andlor physical security (either as corporate policy or warranted by immediate surroundings) Due to the many design and management options that may be dictated by the owner, regulated by local goverment, recommended by Industy groups, or followed as a general practice, iis ot feasible for Tier Standard: Topology and Tier Standard: Operational Sustainabiy to establish citria for these addtional factors {and exposures worldwide. And, the Uptime Institute does not wish to displace or confuse the guidance of local experts, which are key for timely project delivery, cogulatory compliance, and implementation of best practices, For a successtu project, Uptime Insitute recommends thatthe project team create a comprehensive catalogue of project requirements, which Incorporates Tier Standard: Topology, Ter Standard: Operational Sustainabilty, and carefully considered mitigation measures of these adbitional factors and exposures. This approach will ensure the project meets the compliance abjectives of Uptime Insttue’s international standards, as well as local constraints ‘and owner's business case.Uptime Institute - Contents 10 Overview ; 11 Sep . vn 5 12 Papose s = 1.3 References 5 : 14 Fala Peters s - 20. Ter laetenton Dette s : 21 Ter -BaseOaa Ctr Se atu 5 : 22 Ter -Retindt St taste Capt Cnponsis 6 . 25 Ter ll Coney Maina Ste esac 5 24 Ter Fa lt Stee a - 25 Erg Gon Sys 2 - 2.6 Ambient Temperature Design Points 8 ~ 2.7 Communications Routing 8 - 2.8 Makeup Water 9 20 Ter Roques Sina 2 ; 30 Commentary for Appleton oh Tr Standort Topslogy 3 - 2: one Bed Ter Sta 9 - 22 Impact i Dein Coons 9 23 Reston Again Ege Gonos Rin ions (He an a) 24 Ter Frconly Morons © 95 Fecal ore Tar HEN " = 3.6 Non-Compliance Trends 12 Modifications o 12Uptime Institute 1. Overview 1 12 13 14 Scope “This Standard establishes low distinctive defitions of data center st infastrucure Ter classitiations (Tie, Tiel, Tie il, Tier 1), and the performance confirmation tests for determining compliance to ‘he defintions, The Ter classifications describe the sto-evelinfastrucure topology required to sustain data center operations, not the characteristics of individual systems or subsystems. This Standard is predicatac onthe fact hat Gata centers are dependant upon the success and integrated operation of several separate sito Infrastructure subsystems, the numberof which is dependent upon the individual technologies (e9.. power generation, rafigeration, uninterruptible power sources, et) selected to sustain the operation Every subsystem and system intagrated nto the data canter ste niastucture must be consistently deployed with the same site uptime objective to saisly the distinctive Ter requirements Compliance wit the requirements of each Tir is measured by ovtcome-based confirmation tests and operational impacts. This mathod of measurement cifers from a prescriptive design approach or a checklist of required equipment. ‘Commentary on this Standard is na separate section that provides examples for he design and configuration of facilly systems for each Tir topology level. The commentary section also offers guidance in he applieaton and implementation ofthe Tir datintions. in adlton, the commentary section incudes ‘locussion and examples te aid in understanding Tier concepts as wal as information on common design topology short Purpose ‘The purpose ofthis Standards to equip design professionals, data center operators, and non-technical ‘managers wth en objective and effective means for identifying the anticipated performance of citferent data center st infastucture design lopologies References. ‘American Society of Heating, etrigerating, and Ai-Conditoning Engineers, ASHRAE Handbook — Fundamentals (Latest Version). Uptime insttte Fault Tolerant Power Compliance Spoctcation, Version 2.0, Relationship between Tiers and Operational Sustainability Aceredied Tier Designer Tochncal Paper Series Further information can be found at ww uptimeinstiute.com. 2. Site Infrastructure Tier Standards 24 Tier I: Basic Site Infrastructure 2.1.4. The fundamental quirement 4) Tier | basic data center has non-redundant capacity components and a single, rnon-edundantcistoution path serving the crtcal envionment. Ter | infrastructure includes: ‘a dedicated epace for IT Systems; a UPS to iter power spikes, sags, and momentary ‘outages; dedicated cooing equipment; and an engine generator to protect IT functions rom ‘extended poner outages, 1) Twalve hours of on-site fe! storage for engine generators) 21.2 The performance confirmation tests: 2) There sufeient capacity o meet the needs ofthe sit. ») Planned work wil quire most oral of the site infrastructure systems to be shut down atfctng ctical envionment, systems, and end usersUptime Institute 21.8 The operational impacts 4) The sit is susceptible to ciupton from bath planned and unplanned activities. Operation (Human) erors of site infrastructure companenis will cause a data center disruption, 1) An unplanned outage o alae of any capacity system, capaci component, or atnbution ‘element wil impact he crcal environment. ©) The sie infrastructure must be completely shut down on an annual basis to salelypecorm necessary preventive maintenance and repair work. Urgent situations may require more - frequent shutdowns. Falur 1 regularly perform maintenance sigiicanly increases he isk of unplanned disruption as wel asthe severity ofthe consequential flu, 2.2. Tier Il: Redundant Site Infrastructure Capacity Components 2.2.1. The fundamental requtement 2) A Ter data center has redundant capacity components and a single, non-redundant lstrbution path sorving the critical envionment, The redundant components ar extra engine ~ ‘generators, UPS modules and enoray storage, chillers, heat ejection equipment, ump, Coating units, an ue tanks, 'b) Twelve hours of on-site fuel storage for N'capacity ‘22 The pertormance confirmation tests: ‘) Redundant capacity components can be removed trom sence ona planned basis without causing ay ofthe citcal environment to be shut down Removing distibution pats rom service fr maintenance ov other activity requires shutéoun fecal environment ©) There is sufficient permanently installed capacity to meet the needs ofthe sie when Fedurelant components are removed ram series for any reason, » 2.23. The operational impacts: 2) The site is susceptsle to cisuption from both planned activities and unplanned events, (Operation (Human) errors of st intrastucure components may cause a data e center disrupion, ') An unplanned capacity component falure may impact the enical environment. An unplanned outage or falue of any canary system ar diction element wil impact tho entcal environment (©) The site infrastructure must be compltaly shut down on an annual basis safely perorm Preventive maintenance and repair work, Urgent situations may require more frequent shutdowns. Failure to raguary perform maintenance sgniicanty ereases te rik ot unplanned cisruption as well as the severity ofthe consequential fature. 2.3. Tier ll: Concurrently Maintainable Site Infrastructure 2.8.1. The fundamental quirements: 8) Concurrently Maintainable data center has redundant capacity components and multiple = independent cstibution pains soning the ercal envionment. Only one distribution path is requied to serve the eitcal envionment at any te, »)_AIIIT equipments dual powered as defines by Uptime Institute's Fault Tolerant Power ~ ‘Compliance Speciticaton, Version 2 0 and insaled propery to be compat with he {apology ofthe sites architecture. Transter devices, such as point-o-use sultchas, must be Incorporated for ectcal environment that does not meet this specication. Y ©} Twelve hours of on-site fu! storage for N’ capacity ~ 2.32. The pertormance confirmation tests: 8) Each and every capacity component and element in tho distribution paths can be removed from sorvce ona planned basis winout impacting any of the erica environment ) There is suficiont permanently installed capac to meet the needs of the site when redundant components are removed rom service for any reason,2aa Uptimelnstitute @ ‘The operational impacts: 4) The site ls susceptibe to ciruption fem unplanned activities. Operation errors of site infrastructure components may cause a computer disuption. ) An unplanned outage of fature of any capacity system wil impact the ereal environment ©) An unplanned outage or talure ofa eapacty component or distribution element may impact the erica environment 1) Planned st infrastructure maintenance can be performed ty using the redundant capacity ‘components and dstbuton paths ta safely work on the remaining equipment ©) During maintenance activites, the risk of disruption may be elevated. (This maintenance Cendtion does not deteat the Tir rating achieved in normal operations.) 2.4 Tier IV: Fault Tolerant Site Infrastructure at 242 243 “The fundamental coqurements 1) AFaut Tolerant data contr has multiple, independent, physically isolated systems that provide redundant capacity componanis and multiple, independent, verse, active distrbuton paths simutaneousty serving he eiical enonment. The redundant capacity components ‘and diverse stibution pate shal be conigured such that 'N capacity is providing power and ‘cooling to the erica envronment ater any infrastructure failure. 1) AIT equipment is dual powered as defined by Uptime Institute's Fault Toterant Power Compliance Speciicaton, Version 2.0 and installed properly to be compatible withthe topology ofthe st’s architecture. Transler doviees, such as poln-oFuse switchos, must be incorporated for cical environment that dogs not meet his specication (9) Complementary eystems and dstibution paths must be physically isolated rom ane another (compartmentazes) to provent any single event from simultaneously impacting both systems or aistabuton paths. 1d) Continuous Cooling is required. 2} Twelve nous af on-site fel storage for 'N' capaci ‘The pertormance confirmation toss: 1) single faiure of any capacity system, capacity component, or distbution element will not impact the erica evronment b) The infrastructure controls system domonstiatesaulonomous respons toa failure wile sustaining the crieal envitcnment. ©) Each and every capacity component and element inthe distibution paths can bo removed from service ona planned basis without impacting any ofthe cca environment. 18) Thor I eusent capaciy 1o meet the needs ofthe silo whon redundant components oF lstiouion paths are removed rom service for any reason, ‘Tho operational impacts 4) The site isnot susceptible te disxuption fom a single unplanned event The site isnot susceptible to disuption from any planned work acts, } Tho site infrastructure maintenance can be performed by using the redundant capacity ‘components and datbutlon paths to safely work on the remaining equipment. 6) During maintenance acy where redundant capacty components or a distribution path shut down, the cca environment is exposed to an increased isk of disruption in the event a failure oocurs on the remaining path. This maintenance configuration does not defeat the Tier rating achieved in normal operations. 12) Operation ofthe fre alarm, te suppression, or the emergency power off (EPO) feature may cause a data contor disruption.