USO097 13290B2

(12) United States Patent (10) Patent No.: US 9,713,290 B2

James et al. (45) Date of Patent: Jul.18, 2017
(54) DATACENTER IMMERSED IN COOLING H05K 7/20736. H05K 7/20218: H05K
LIQUID 7/20709; H05K 7/1497; H05K 7/20;
H05K 7/20336; H05K 7/20381; H05K
(71) Applicant: MICROSOFT TECHNOLOGY 7/20345; H05K 7/207 18: H05K 5/067;
LICENSING, LLC, Redmond, WA H05K 7/20645; G06F 1/20: G06F
(US) 22OOf2O1
(72) Inventors: Sean James, Olympia, WA (US); Todd See application file for complete search history.
Robert Rawlings, Sammamish, WA (56) References Cited
(US)
U.S. PATENT DOCUMENTS
(73) Assignee: Microsoft Technology Licensing, LLC,
Redmond, WA (US) 1,757,174 A 6/1930 Douglas
2,870,729 A 1/1959 Shannon et al.
(*) Notice: Subject to any disclaimer, the term of this (Continued)
patent is extended or adjusted under 35
U.S.C. 154(b) by 26 days. FOREIGN PATENT DOCUMENTS

(21) Appl. No.: 14/320,019 CN 2O1846435 U. 5, 2011

DE 102011 115657 A1 3, 2013
(22) Filed: Jun. 30, 2014 (Continued)
(65) Prior Publication Data OTHER PUBLICATIONS
US 2015/0382515 A1 Dec. 31, 2015 “International Search Report & Written Opinion Received for PCT
(51) Int. Cl. Application No. PCT/US2015/037848, Mailed Date: Oct. 6, 2015,
10 Pages.
HOIL 23/2473 (2006.01)
H05K 7/20 (2006.01) (Continued)
HOIL 23/2427 (2006.01)
(52) U.S. Cl. Primary Examiner — Adam B Dravininkas
CPC ....... H05K 7/20818 (2013.01); HOIL 23/427 (57) ABSTRACT
(2013.01); H0IL 23/473 (2013.01); H05K
7/203 (2013.01); H05K 7/2079 (2013.01); The subject disclosure is directed towards a datacenter or
H05K 7/20327 (2013.01); H05K 7/20345 partial datacenter (e.g., a datacenter module) contained in a
(2013.01); H05K 7/20809 (2013.01) sealed container. The container may be filled with a cooling
(58) Field of Classification Search fluid, such as a dielectric fluid, to help cool the datacenter
CPC ......... H01L 2924/0002; H01L 2924/00; H01L components. The container and its internal datacenter or
2924/00014; H01L 23/473; H01L 23/427; datacenter portion may be submerged in water, in which
H01L 23/467; H05K 7/20772; H05K event the fluid also helps to equalize the external water
7/20809; H05K 7/20836; H05K 7/20781; pressure.
H05K 7/2079; H05K 7/203; H05K
7/20236; H05K 7/20818; H05K 7/20281; 11 Claims, 8 Drawing Sheets
Tow Fron
 US 9,713.290 B2
Page 2

(56) References Cited U.S. Appl. No. 13/917,636, Rubenstein, et al., “Renewable Energy
Based Datacenter Cooling', filed Jun. 13, 2013, 31 pages.
U.S. PATENT DOCUMENTS Miller, Rich, “Google Planning Offshore Data Barges'. Published
on: Sep. 6, 2008, 5 pages Available at: http://www.
4411,213 A 10, 1983 Laukien datacenterknowledge.com/archives/2008/09/06/google-planning
5,511,504 A 4, 1996 Martin offshore-data-bargest.
6,186,702 B1 2/2001 Bartkowski
7,525,207 B2 4/2009 Clidaras et al. Miller, Rich, "Data Centers on Cargo Ships?”. Published on: Jan. 8,
8,549,869 B1 10/2013 Whitted et al. 2008, 5 pages Available at: http://www.datacenterknowledge.com/
8,636,565 B2 1/2014 Carlson et al. archives/2008/01/08/data-centers-on-cargo-ships/.
8,853,872 B2 10/2014 Clidaras et al. “International Search Report & Written Opinion Received for PCT
8,854,809 B2 10/2014 Neumann et al. Application No. PCT/US2015/037847, Mailed Date: Nov. 9, 2015,
8,913,383 B1* 12/2014 Goldsmith ......... HOSK 7,20236 11 pages.
257/714
2008/0055850 A1 3f2008 Carlson ..................... G06F1 20 Trabish, Herman K. “Ocean Energy to Power Google's Sea-Going
361/695 Data Center'. Published on: Sep. 10, 2008, 4 pages Available at:
2008/0209234 A1* 8, 2008 Clidaras .............. FO3B 13, 1885 http://newenergynews.blogspot.in/2008/09/ocean-energy-to-power
T13,300 googles-Sea-going.html.
2009/00784.01 A1 3/2009 Cichanowicz Offce Action Summary, U.S. Appl. No. 14/272,656, Notification
2009, 02291.94 A1 9, 2009 Armillas Date: Nov. 30, 2015, 12 pages.
2009,029.5167 A1* 12, 2009 Clidaras .................. FO3B 13/20 Green (Low Carbon) Data Center Blog. The Under Water Data
290/55 Center, Response to Risks of Google's Floating Data Center,
2009,0297.270 A1 12, 2009 Black et al. Submerge, Sep. 12, 2008, 3 pages. Available at: http://www.
2011 0132579 A1* 6, 2011 Best ................... HOSK 7/2O763
165,104.31
greenm3.com/gdcblog/2008/9/12/the-under-water-data-center-re
2011/0154842 A1* 6/2011 Heydari ............. HOSK 7,20836 sponse-to-risks-googlersquo.html.
62.259.2 U.S. Appl. No. 14/272,656, Dehlsen, “Marine Subsurface Data
2011/0240497 A1 10/2011 Dechene et al. Center Vessel', filed May 8, 2014, 16 pages.
2011/0247348 A1* 10/2011 Mashiko .............. HOSK 7.2079 “International Preliminary Report on Patentability Issued in PCT
62/62 Application No. PCT/US2015/037848, Mailed Date: Apr. 21,
2012fO232879 A1 9/2012 Iyengar et al. 2016, 6 Pages.
2013/0044426 A1 2/2013 Neumann ................. G06F1 20 Matteson, et al., “Maximizing Data Center Energy Efficiency by
361? 679.54 Utilizing New Thermal Management and Acoustic Control Meth
2013,0058O29 A1 3/2013 Ootani et al. odology”. In International Technical Conference and Exhibition on
2013/O125825 A1 5/2013 Kania et al. Packaging and Integration of Electronic and Photonic
2013. O190941 A1 7/2013 Cader et al. Microsystems, vol. 2, Jul. 16, 2013, 1 page.
2015,0194813 A1* 7, 2015 Finn ..................... HOSK 7, 1497 Masaki, et al., “Underwater Surveillance System to Counteract
307/19 Associated Underwater Threats'. In NEC Technical Journal, vol. 8,
2015,0321739 A1* 11/2015 Dehlsen ................... B63G 8.18
165/45
Issue 1, Sep. 2013, pp. 63-67.
Felemban, Emad, “Advanced Border Intrusion Detection and Sur
veillance Using Wireless Sensor Network Technology”. In Interna
FOREIGN PATENT DOCUMENTS tional Journal of Communications, Network and System Sciences,
WO 2009108070 A1 9, 2009
vol. 6, Issue 5, May 2013, pp. 251-259.
WO 20121296.12 A1 10, 2012 Laumer, John, “Google Floats a Data Center Patent: Offshore,
WO 2013184820 A1 12/2013 Ocean-Cooled, Wave-Powered, and Modular'. Published on: Sep.
WO 2014 120275 A1 8, 2014 20, 2008, 4 pages. Available at: http://www.treehugger.com/corpo
rate-responsibility/google-floats-a-data-center-patent-offshore
ocean-cooled-wave-powered-and-modular.html.
OTHER PUBLICATIONS Fletcher, Joanna, “What Happened to Google's Floating Data
Center?". Published on: Dec. 4, 2010, 2 pages. Available at:
Patel, et al., “Thermal Considerations in Cooling Large Scale High http://www.hostway.com/web-resources/find-web-hosting/what
Compute Density Data Centers'. In the Eighth Intersociety Con happened-to-googles-floating-data-center?.
ference on Thermal and Thermomechanical Phenomena in Elec
tronic Systems, May 2002, 10 pages. * cited by examiner
U.S. Patent Jul.18, 2017 Sheet 1 of 8 US 9,713,290 B2

111111
Heat Transfer
Subsystem (e.g.,
Coils, Pumps, Fans
and/or Natural
Circulation)
106

104
DataCenter ServerS /
Storage Network

Dielectric Fluid-Filled COntainer

F.G. 1
U.S. Patent Jul.18, 2017 Sheet 2 of 8 US 9,713,290 B2

Water
PreSSure

Water
Pressure

DataCenter ServerS / Water

Storage Network Pressure

Dielectric Fluid-Filled Container

FIG. 2
U.S. Patent Jul.18, 2017 Sheet 3 of 8 US 9,713,290 B2

304
U.S. Patent Jul.18, 2017 Sheet 4 of 8 US 9,713,290 B2

404

Component(s)
440

Fluid CollectOr

FIG. 4
U.S. Patent Jul.18, 2017 Sheet S of 8 US 9,713,290 B2

504

Fluid Tank
554

W y f
1
() V Dripper
()
552
Vapor Vapor
Fluid

w w
M
f
y f

Component(s)
550

Pump
Fluid Collector 58 556

FIG. 5
U.S. Patent Jul.18, 2017 Sheet 6 of 8 US 9,713,290 B2

TO POWer TO / Fron
SOurce USerS

FIG. 6
U.S. Patent Jul.18, 2017 Sheet 7 of 8 US 9,713,290 B2

FIG. 7

ASSemble DataCenter
702 Hardware Components
into Container

704 Seal Container except for

Fluid Inlet, Air Outlet

Optionally Move
7O6 s
TN. Unfied Container
to Deployment Site
as as as so a as . . . .

Fil Container With

708 Dielectric Fluid, Seal Inlet
and Outlet

710 Perform any Final Tests /

Inspections

712 Deploy Datacenter

Container(s)

714 Service Users

U.S. Patent Jul.18, 2017 Sheet 8 of 8 US 9,713,290 B2

O
 US 9,713,290 B2
1. 2
DATACENTER IMMERSED IN COOLING FIG. 8 is a block diagram representing an example
LIQUID datacenter cooled by circulating gas/fluid containing/com
puting device into which one or more aspects of various
BACKGROUND embodiments described herein can be implemented.
As cloud-based computing and cloud-based services DETAILED DESCRIPTION
grow, datacenters need to be provided to serve client cus
tomers. Customers want fast speeds (lowest possible Various aspects of the technology described herein are
latency) for their cloud applications. In order to satisfy generally directed towards cooling a datacenter (generally a
customers, future data centers need to be positioned as close 10 facility comprising centrally managed computing resources
as possible to the customer base. At the same time, consid and related Support systems) or a partial datacenter (such as
eration needs to be given to privacy, security, environmental a modular component that with other modules provides a
conditions, real estate availability, access to power, cost of datacenter), with a dielectric fluid. In one or more aspects,
power and so on. the datacenter may be filled with the fluid, which may be
15 circulated for cooling purposes. In alternative aspects, the
SUMMARY
fluid may be applied to (e.g., sprayed or dripped on) the
components which then boils the fluid into a vapor that gets
cooled back into a fluid.
This Summary is provided to introduce a selection of Further, the datacenter may be designed to be submerged,
representative concepts in a simplified form that are further for example on the ocean floor or the floor of any similar
described below in the Detailed Description. This Summary body of water, Such as a lake, river, flooded former quarry
is not intended to identify key features or essential features and so on. In this way, the datacenter may be deployed
of the claimed subject matter, nor is it intended to be used relatively close to current and potential customers, and
in any way that would limit the scope of the claimed subject positioned in a way to take advantage of Sustainable power
matter. 25 that is also environmentally friendly, and take advantage of
Briefly, one or more of various aspects of the subject the massive heat sink provided by the water.
matter described herein are directed towards a datacenter or If the datacenter is filled with the fluid, the fluid will act
partial datacenter (e.g., a datacenter module) immersed in a to equalize the water pressure, allowing the datacenter to be
cooling fluid, which may be a dielectric fluid. A container Submerged in very deep water without needing a particularly
Surrounds and seals the cooling fluid and datacenter or 30 strong (thick metal) hull. As a result, the hull may be
partial datacenter, and a heat transfer Subsystem cools the constructed of any material that is resistant to corrosion and
cooling fluid and the datacenter or partial datacenter. may have other desirable properties, e.g., acts as a good heat
The container may be submerged in water to provide an conductor, Such as certain polymers that would be crushed
underwater datacenter. If so, the internal fluid also acts to at deeper pressures if not (Substantially) equalized. By
equalize or Substantially equalize the water pressure. 35 positioning the datacenter in deep water, such as anchoring
Other advantages may become apparent from the follow it or sinking it to the ocean floor, the risks of umbilical
ing detailed description when taken in conjunction with the detachment or damage to the datacenter by external forces
drawings. are significantly reduced.
It should be understood that any of the examples herein
BRIEF DESCRIPTION OF THE DRAWINGS 40 are non-limiting. For example, in one aspect Submerged
datacenters are exemplified, as is the concept of positioning
The present invention is illustrated by way of example datacenters on or near the water floor, e.g., by sinking them.
and not limited in the accompanying figures in which like Any body of water including an ocean, riverbed, sea, lake,
reference numerals indicate similar elements and in which: deep pond, and so on may be used. However, Submersion of
FIG. 1 is an example representation of a center or partial 45 a datacenter, or any part thereof (e.g., a module), is only one
datacenter (e.g., a datacenter module) in which a dielectric way to use the technology described herein. As such, the
fluid is used to help cool the datacenter components, accord present invention is not limited to any particular embodi
ing to one or more example implementations. ments, aspects, concepts, structures, functionalities or
FIG. 2 is an example representation of Submerged data examples described herein. Rather, any of the embodiments,
center or partial datacenter, in which fluid therein acts to 50 aspects, concepts, structures, functionalities or examples
equalize the external water pressure, according to one or described herein are non-limiting, and the present invention
more example implementations. may be used in various ways that provide benefits and
FIG. 3 is an example representation of how a cooling fluid advantages in datacenters and computing in general.
may be used to transfer heat away from a datacenter or As generally represented in the example implementation
datacenter module, according to one or more example imple 55 of FIG. 1, a datacenter 102 (or partial datacenter, such as a
mentations. module that may be coupled to other modules to increase a
FIGS. 4 and 5 are block diagrams each representing an datacenter's capacity), is placed in a container 104 filled
example of how a dielectric fluid may be used to cool with a dielectric fluid. The fluid may be circulated by any
datacenter components, according to one or more example Suitable heat transfer Subsystem 106, e.g., using pumps that
implementations. 60 in a closed loop system pump the fluid through coils or the
FIG. 6 is an example representation of a modular data like, by natural circulation (e.g., heat pipe technology)
center having chamfered edges to facilitate sealing of a resulting from heat rising/fluid vaporizing at a desired
cooling fluid therein, and strength, such as for Submersion, temperature and pressure (or where a fan can move and cool
according to one or more example implementations. the vapor), and so on.
FIG. 7 is a flow diagram comprising example steps that 65 The container 104 and (any external part of) the heat
may be used in immersing a datacenter in a cooling fluid, transfer subsystem 106 is sealed so that the fluid cannot
according to one or more example implementations. escape. It should be noted that the container surface 104 also
 US 9,713,290 B2
3 4
may transfer heat away from the datacenter, and if internal FIGS. 4 and 5 show examples in which fluid cooling may
circulation is Sufficient, an external heat transfer Subsystem be used in a different way, namely as a dielectric fluid
may not be needed. Notwithstanding, in one or more imple applied in its liquid state to a component 440 or 550, such
mentations, the container is designed to be submerged, and as sprayed by a sprayer 442 (FIG. 4) or dripped by a dripper
it may be desirable to externally locate any pumps, fans and 552 (replenished from a tank 5.54) onto a microprocessor or
so on that include moving parts to facilitate replacement or heat sink coupled thereto, for example, or any group of
repair if needed. components. When the fluid hits the component(s) 440 or
Various deployments and other aspects of Submerged 550, the heat of the component(s) vaporizes the liquid. The
containers are described in copending patent application 10
vapor cools back to a liquid, e.g., via the cooled walls of a
entitled “Submerged Datacenter” (U.S. Application No. container 404 or 504, which then drips into a collector 448
14/320,019), including ways to obtain power from the water or 558. The sprayer 442 may pump from the collector 448
for powering the datacenter, at least in part. For example, as in FIG. 4, or a pump 556 or the like may be used to pump
waves, tides, and currents may be used to generate power the fluid back into the tank 5.54 as in FIG. 5. Any fluid that
from the water, power also may be provided via waterfalls 15 is not vaporized, e.g., one that hits a relatively cool com
and wind turbines, including for Submerged datacenters. ponent, simply drips into the collector 448 or 558. Note that
Further, as can be readily appreciated, the heat transferred an actual fluid collector need not be used, as the bottom of
from the datacenter may be used to generate some power. the container 404 or 504 can act as a collector.
FIG. 2 shows the concept of how the cooling fluid also FIG. 6 shows a datacenter 660 comprised of a number of
helps with pressure equalization of a container 204. Because (e.g., five) datacenter modules that are arranged in an array,
fluids cannot be (significantly) compressed, the container such as deployed on the ocean floor. In the example of FIG.
204 need not be designed for the significant amount of water 6, each module’s housing serves as the fluid container,
pressure that occurs at deeper depths (although the design of although it is feasible for the module housing to be put into
the components of the datacenter or partial datacenter 202 a separate container. Each of the modules is connected to
will need to take the container's internal pressure into 25 power and to a communication medium Such as fiber optics
consideration). This allows for container materials that are for sending and receiving data communications. Note that
easy to seal for Submarine conditions, which may be highly any or all of the modules may be coupled to each other for
corrosive in Saltwater deployments. For example, polymers communications within the datacenter.
may be used for the container, or as a coating on a relatively The exemplified modules each include chamfered edges
thin metal container, rather than needing thick metal for its 30 which in general, provides for better strength and sealing
strength. properties.
The dielectric fluid enables a high-density server, storage As described herein, the modules may be filled with
and network equipment design by providing very efficient dielectric fluid and deployed. The modules may be deployed
cooling; immersion liquid cooling and conduction is over underwater to a depth and/or location where they are rea
twenty times more efficient as a heat rejection medium than 35 sonably secure (the containers would not be able to be
air. Suitable dielectric fluids are readily commercially avail opened underwater, at least not easily).
able. Mineral oil is one such fluid that is not particularly The water also provides a heat sink for conductive trans
harmful to the environment should a leak occur. Carbon fer of heat from the module’s components to the internal
dioxide turns to a liquid at pressures and temperatures that fluid, and thereby to the surrounding water. For example,
may exist at Some deployment locations. A mixture of fluids 40 heat pipe technology or a hydronic loop may be used to draw
that provide the desirable properties may be used. heat from the hardware containing-tank through heat sink
When Submerged, the immersion cooling fluid and con coils that are cooled by the outside water outside. In this
sistent heat sink from the outside sea water temperature may way, the cooling system rejects heat from the servers to the
be used to keep the electrical components that are running surrounding body of water. The heat transfer in these medi
inside the container within a very low and narrow operating 45 ums are very efficient with respect to keeping the change in
envelope. The rate of circulation may be controlled if temperature (delta T) Small, which can minimize the impact
needed, however because relatively deep water does not to the Surrounding ecosystem.
change in temperature very much, the design of the heat Further, datacenter components, which are sensitive to
transfer Subsystem can be such that circulation control is not corrosive contamination, are protected by the dielectric
needed, or at least not as much compared to air cooling or 50 fluid. Indeed, hardware in marine environments has a high
cooling at shallower depths where the external ambient failure rate from salt in the air and humidity.
temperature can significantly vary. This will improve the With respect to construction and deployment, FIG. 7 is a
hardware reliability, and to an extent the hardware's oper flow diagram showing example steps; (note that at least
ating efficiency. Some of the example steps need not occur in the order
For non-Submerged deployments, or deployments in rela 55 shown). In the example of FIG. 7, containers which may be
tively shallow water, a sealed datacenter 302 (e.g., each separate modules are used as examples, however it is
module) may still be cooled by circulating fluid around it, understood that an entire datacenter may be contained in
even if the datacenter 302 itself is not filled with fluid, as container or module, and thus a single “module' may serve
generally represented in FIG. 3. If the datacenter or partial as a datacenter. Indeed, where population/demand for cloud
datacenter 302 is sealed within a Sufficiently strong housing, 60 services is not too great, a single module (as thought of in
the entire container 304 may be submerged into deeper and the conventional sense) may suffice to serve a region.
deeper water. Although this is likely far less efficient (e.g., Step 702 represents installing the datacenter hardware
in terms of container size, component density, rate of cir components. Because of the immersive cooling, in general,
culation and so on) relative to immersing a datacenter and its these components may be much more densely arranged,
components in the fluid, a dielectric fluid is not needed in the 65 both within each server/storage unit and with respect to one
example of FIG. 3, which may be a factor in certain another. This allows the datacenter to be compact and
situations. efficient, which is cost effective (from both a water-tight
 US 9,713,290 B2
5 6
enclosure perspective and volume of dielectric fluid per gation switches 806-806. In this way, each server may
spective), as the larger the enclosure and the more fluid communicate with any other server, including a server in a
needed, the higher the cost. different rack. Note that in this example, a higher-level
Step 704 seals the container, except for the dielectric fluid aggregation Switch 808 couples the rack-level aggregation
inlet (and any gas outlet, if needed, such as to allow air to switches 806-806, and there may be one or more additional
escape while filling). The container may then be transported levels of aggregation Switch couplings.
(step 706) to its deployment site before it is filled, e.g., so it As represented in FIG. 8 by the rounded arrows, the
is lighter, meaning it can be floated (at least more easily) for exemplified datacenter has gas and/or dielectric fluid circu
towing/water deployments, and so forth. lated throughout, which may be via pumps, fans and/or
Step 708 represents filling the container with the fluid 10
natural circulation. A heat transfer subsystem 810, which
(which may be a gas at Surface pressures and temperatures), may use coils, radiators, fluid pumps, fans and so forth
and sealing the inlet/outlet ports. If a large amount of fluid transfers heat away from the datacenter/module to the sur
is needed, the fluid may be transported via a tanker, for rounding water and/or for use in power generation. Note that
example. At this time, the container and its components are a modules or datacenter's hull itself may be used as a heat
ready for deployment, (after performing any final tests or 15
transfer mechanism.
inspections (step 710) that may be desirable to perform
before deployment, particularly if sinking the datacenter in CONCLUSION
water). Note that it is feasible to move containers after
filling, for example if surface conditions at the desired
location makes it difficult to fill the container there. While the invention is susceptible to various modifica
Step 712 represents deployment, which includes attaching tions and alternative constructions, certain illustrated
any external cabling to the modules, and if in water, sinking embodiments thereof are shown in the drawings and have
the containers. Step 714 represents using the datacenter to been described above in detail. It should be understood,
service users. Note that adding the dielectric fluid increases however, that there is no intention to limit the invention to
the weight of each container, and thus previously floating 25 the specific forms disclosed, but on the contrary, the inten
containers may now be able to sink into water by design. tion is to cover all modifications, alternative constructions,
Multiple containers may be arranged on a platform, with the and equivalents falling within the spirit and scope of the
platform then Sunk, assisted by the increased weight of the invention.
containers. In addition to the various embodiments described herein,
Fluid-filled datacenters are intended to be unmanned, at 30 it is to be understood that other similar embodiments can be
least largely. This is particularly significant for Submerged used or modifications and additions can be made to the
datacenters, which may be located many (e.g., twenty-four) described embodiment(s) for performing the same or
miles offshore and on the ocean floor, and thus may be equivalent function of the corresponding embodiment(s)
extremely difficult to service. Thus, the design is that the without deviating therefrom. Still further, multiple process
datacenter is “unserviceable,” with equipment and hardware 35 ing chips or multiple devices can share the performance of
expected to not fail during active usage. However, the use of one or more functions described herein, and similarly,
modules allows for rotating new modules into a datacenter storage can be effected across a plurality of devices. Accord
as older ones are removed, e.g., detached and floated up for ingly, the invention is not to be limited to any single
recycling. embodiment, but rather is to be construed in breadth, spirit
40 and scope in accordance with the appended claims.
Example Datacenter Environment
What is claimed is:
One of ordinary skill in the art can appreciate that the 1. A system comprising:
various embodiments and methods described herein can be a datacenter or partial datacenter immersed in a cooling
implemented in connection with any number of hardware 45 fluid, the data center or partial datacenter comprising:
devices, which can be deployed as part of a datacenter or a sealed container that Surrounds and seals the cooling
other computing environment, and can be connected to any fluid and the datacenter or partial datacenter, the
kind of data store or stores. Thus, the technology is not sealed container configured for Submersion in Sub
limited to a datacenter in the conventional sense, but may be marine conditions, including having chamfered
used in any situation where computing power is needed near 50 edges; and
a certain location and heat dissipation is a consideration. aheat transfer Subsystem configured to cool the cooling
FIG. 8 shows an example submerged datacenter 800 (or fluid and the datacenter or partial datacenter and
one datacenter module) that is exemplified as having a transfer heat away from the datacenter or partial
tree-like topology. A plurality of racks 802-802, each have datacenter to Surrounding water in the Submarine
servers, which communicate through a top of rack Switch 55 conditions, including to circulate a dielectric fluid
804-804. The servers may include storage, or at least part throughout the sealed container via a sealed circu
of the storage may be separately located. A typical network lation pipe passing through the sealed container and
has twenty to forty servers per rack, with increasingly to transfer heat from the circulated dielectric fluid to
powerful links and Switches going up the tree. Note that the Surrounding water via a heat transfer mechanism.
datacenters are not limited to tree-like topologies, but can be 60 2. The system of claim 1 wherein the cooling fluid is a
used in any topology. A small amount of the computing dielectric fluid, and wherein components of the datacenter
power may be used to monitor the Submerged datacenter are immersed in the dielectric fluid.
sensors, run any fans, pumps and so on, operate an active 3. The system of claim 1 wherein the container incorpo
leveling system, and so on, although this may be done with rates or is coupled to at least one pump to circulate the
separate machine logic. 65 cooling fluid.
As represented in FIG. 8, each top of rack switch 804 4. The system of claim 1 wherein the sealed container is
804, is coupled to one another through one or more aggre Submerged in water, and wherein the heat transfer Subsystem
 US 9,713,290 B2
7 8
is configured to cool the cooling fluid and the datacenter or reduce or equalize a pressure difference between exter
partial datacenter by transferring at least some heat from the nal water pressure external to the sealed container and
sealed container to the water. at least part of an internal pressure of the sealed
5. The system of claim 4 wherein the container is suffi container and to transfer the heat away from the sealed
ciently filled with the cooling fluid to reduce or equalize a container into the water.
pressure difference between external water pressure and at 8. The system of claim 7 wherein the heat transfer
least part of the internal container pressure. Subsystem comprises a pump that circulates the cooling fluid
6. The system of claim 4 wherein the heat transfer within the sealed container.
Subsystem comprises heat pipe technology that provides 9. The system of 7 wherein at least part of the heat transfer
conductive transfer of heat from the cooling fluid within the 10 Subsystem uses a surface of the sealed container to transfer
sealed container to the water surrounding the sealed con at least some heat to the water in submarine conditions
tainer. outside the sealed container.
7. A system comprising: 10. The system of 7 wherein the heat transfer subsystem
a datacenter or partial datacenter enclosed in a sealed comprises a hydronic loop that provides conductive transfer
container comprising chamfered edges and submerged 15 of heat from the cooling fluid within the sealed container to
in water in submarine conditions, the sealed container the water surrounding the sealed container.
further comprising: 11. The system of 7 wherein at least part of the heat
a heat transfer subsystem that transfers heat away from transfer subsystem uses at least one heat pipe to cool the
the sealed container into the water, including by circu datacenter or partial datacenter and the cooling fluid.
lating a cooling fluid throughout the sealed container to