Immerse Yourself in the Cooling Waters

Disclaimer #1: DO NOT USE WATER IMMERSION TO COOL YOUR COMPUTER OR SERVER!!!
Disclaimer #2: I WORK IN SALES, ANY TECHNICAL INFORMATION CONTAINED HEREIN CANNOT BE TRUSTED.

Properly cooling systems has always been an important topic for me… early on in my career, I encountered a thermal hurdle to overcome and since then keeping systems cool has been near and dear to my heart. It all started when I was working for a small server startup many years ago. One of my first sales there was for a small batch of 1U’s deployed for a government project. After the customer received these systems, I got the dreaded call that there was a problem, every system was overheating. This seemed impossible to me, all the systems were tested, burned-in and working perfectly at our facility. How could this happen?

Not to tire you with all the details… but next came a prolonged torture:

Getting the systems back to our facility for rework.
Shipping them again to the customer and getting a second dreaded call that they were still overheating.
Getting the systems back to our facility for rework.
Shipping them again to the customer and getting a third dreaded call that they were still overheating.

It took some time to find the root cause, a perfect storm was occurring where our system was installed with another manufacturer’s gear (they didn’t play well together) into a data center with terrible airflow and cooling.

In the end, the problem was solved by gaining access to the secure facility that housed all this gear, reorganizing a few racks and replacing all the servers with systems that were properly engineered.

To my customer from this story… thank you for your extreme patience and for the many subsequent orders. Sometimes how you handle a problem, shows value in of itself.

In recent years I have witnessed an increased interest in alternative cooling methods for compute. As compute density increases, floor space decreases and/or the wattage of components grows, the need to find more efficient means to cool systems has become more prevalent. Historically, systems have been cooled with air using front to back airflow: fans pull cool air through the front of a chassis and exhaust hot air from the back. Data centers typically plan accordingly and position racks in a cold and hot aisles orientation, air conditioners would vent cold air into the “cold” aisle, the servers would heat that up and exhaust into a hot isle… where that hot air would be contained, exhausted, returned to the cooling unit or even allowed to dissipate into the ambient air. All inefficient methods, but they worked.

When I first heard about vertical cooling, that seemed revolutionary. In this model, instead of a traditional front to back airflow, things are simply reoriented to flow from bottom to top. This made complete logical sense, hot air rises, why would anyone mess with the natural flow to begin with? In the case of the Cirrascale BladeRack solution, a rack of servers can be placed over a cold vent tile, air flow is contained to the rack itself and exhausted out the top. More efficient all around and no, this is not a cheap marketing ploy by me to boast of our superior cooling product, but rather relevant to the discussion and a stepping stone to the topic I’d like to really get to… immersion cooling.

Currently, the predominant method of cooling systems is air and making progress is liquid cooling. Liquid cooling can take on a few shapes, two that I am familiar with. 1. Liquid is run to specific heat sources (such as the CPU) through pipes that run to a radiator, where fans are still employed to remove heat. Air still plays a major role and is needed to pass through the chassis to cool other components. 2. Fans are removed from the system (all moving components are removed) and the complete system is immersed in non-conductive liquid which dissipates the heat from every component.

At first, the concept seems too far out there for me. From a serviceability standpoint alone, I can’t imagine dealing with all that. However, in dealing with servicing a system, once you rule out failures that are caused by overheating, then remove any early infant mortality, most systems that are built from quality components will run in a production environment without fail for an extended amount of time. So, dealing with all that could be minimal at first, or one would hope.

Serviceability aside, the cooling advantages alone are intriguing and outstanding! You no longer need to worry about air flow, the liquid fully engulfs every component and is in complete contact with every heat source, dissipating heat immediately. What does this mean to me? It means that the design of systems can start to grow more unique and more dense… unfortunately the laws of thermodynamics still apply, but now we can get more creative and aren’t as hindered by the placement of components that would traditionally block airflow. These systems still take on a 19” U form factor and an upright rack is now replaced by 19” U bath.

I’ve been fortunate enough to be involved in some conceptual designs for immersion cooling which I’ve displayed below  and found that working with immersion cooling is very cool (no pun intended.)  There is just something that I find enjoyable in the process of turning imagination into a reality… we do it quite often at Cirrascale with our blades. I’ve been involved in projects, such as desktop consolidation that took what would be a rather large workstation and condensed it into nothing more than a 1U size blade and cigar boxed sized device at your desk.

Immersion cooling… now we’re operating on a whole new level. Aside from the heat transfer and thermal advantages, the data center can now be relatively quiet. Makes me wonder where technology is heading, what will be the next method to cooling systems. Many years ago completely immersing a system in liquid may have seemed completely outlandish, but now it doesn’t seem so crazy.

To anyone reading who is currently using immersion cooling, please reach out to me and tell me more. I would love to know how you’re using immersion and how it works for you, if anything for my own edification.

Thank you for reading my post.

1U POC Single/Dual Processor system with x16 PLX Riser (for four GPU/Phi Cards)

1U POC Single/Dual Processor system with x16 PLX Riser (for four GPU/Phi Cards)

3U POC Four independent Single/Dual Processor system each with a x16 PLX Riser  (for four GPU/Phi Cards per independent system= 16 GPU/Phi cards in 3U)

3U POC Four independent Single/Dual Processor system each with a x16 PLX Riser
(for four GPU/Phi Cards per independent system= 16 GPU/Phi cards in 3U)

4U POC Single/Dual Processor system with four x16 PLX Risers  (for sixteen GPU/Phi Cards per system)

4U POC Single/Dual Processor system with four x16 PLX Risers
(for sixteen GPU/Phi Cards per system)

 

Share this Post: Facebook Twitter Pinterest Google Plus StumbleUpon Reddit RSS Email

1 Comment

  1. First and overall, I loved the Disclaimers! About cooling, there are so many details and tweaks that can be done, you mentioned about the resident radiators used in some of our HPC blade design, how about the rack level liquid cooled running in and out each blade (pretty clever)… and with airflow alone we are working on finding way to even improve our current VCT. Thanks from bringing the awareness and importance of the cooling and the Immersion technology to the subject, so people still not believe this is being done.

Leave a Comment

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Copy this code

and paste it here *