Microsoft Trialing Underwater Data Center/Server Farm

When it comes to setting up a facility to host a large number of servers, three of the biggest considerations when building a site are access to power, access to telecom data lines, and cooling. For the third item, cooling, it is a major concern, due to the massive amounts of heat that large numbers of computers generate. It's not an option to install a bunch of server racks into an enclosed area (like a warehouse), and call it day. Most data centers have massive sets of ducts and ventilation shafts, and very large and powerful airflow control systems to actively remove the heat generated from large numbers of computers concentrated in one area.

We have seen some novel ideas to try to tap into environmental conditions to aid in cooling data centers. As the cryptocurrency craze has escalated, groups setting up large scale crypto "mining" operations have set up shop in Iceland, hoping to capitalize on the island's lower temperature climate for lowering their cost in cooling their server farms.

Microsoft has taken this notion a step further: they have been trialing operation of a data center submersed below the ocean:

Two years ago, Microsoft sank a data centre off the coast of Orkney in a wild experiment.
That data centre has now been retrieved from the ocean floor, and Microsoft researchers are assessing how it has performed, and what they can learn from it about energy efficiency

Their first conclusion is that the cylinder packed with servers had a lower failure rate than a conventional data centre.
When the container was hauled off the seabed around half a mile offshore after being placed there in May 2018, just eight out of the 855 servers on board had failed. That compares very well with a conventional data centre.

"Our failure rate in the water is one-eighth of what we see on land," says Ben Cutler, who has led what Microsoft calls Project Natick.
The team is speculating that the greater reliability may be connected to the fact that there were no humans on board, and that nitrogen rather than oxygen was pumped into the capsule.

Orkney was chosen for the trial by Microsoft, partly because it was a centre for renewable energy research in a place where the climate was temperate - perhaps even chilly. The idea was that the cost of cooling computers would be lower if they were under water.

I think we will see more and more of this kinds of ventures, where data centers are established in remote areas where environmental conditions allow for unique ways to cool the facility in cost efficient ways. Of course, this necessitates access to power and importantly network interconnects, but it's seeming like projects to stand up much higher bandwidth wireless network systems (Starlink comes to mind) might someday permit data centers to exist in places where running physical cable lines is very difficult.

I think that we will start seeing exploration of near-orbit server farms in the not distant future, and I have some thoughts around it that I want to explore in another post.

Underwater server farms in northern latitudes has a time limited cost effectiveness on financial returns. Meaning it will only work so long as the lower temperatures hold. That may well be for a couple of hundred years, but true corporations plan much further out.

Again there is the issue of dumping server farms on possible archeology sites.

Planning longterm newer networks underwater may be a placeholder, but the reality is we need to design better systems than the fragile spiderweb we all take for granted currently. Spending billions on a system that depends on the current chips an wires makes little financial sense, unless Microsoft an others have tech we're unaware of they're going to roll out.

originally posted by: Caver78
That may well be for a couple of hundred years, but true corporations plan much further out.

If you think corporations have a multi-century strategic plan you're kidding yourself.

Cue the inane 'duh Rothschilds' reply.

These sort of DC's are designed to have zero human interaction once they are up and running.

Given the rise in general CPU efficiency a 3-5 year life span before the old one is pulled out and the new one dropped in will be the norm and they're designed with redundancy so failure isn't important as there will be other servers in the DC and also the risk of catastrophic failure will be that it will be covered for handing over workloads at any time.

The cost savings of cooling cannot be underestimated as depending on the location it can easily end up adding 10-20% on the operating cost.

originally posted by: AugustusMasonicus

originally posted by: Caver78
That may well be for a couple of hundred years, but true corporations plan much further out.

If you think corporations have a multi-century strategic plan you're kidding yourself.

Cue the inane 'duh Rothschilds' reply.

I was about to say some companies may have a research and development department,but centuries in the future? Flexibility corporately is the buzzword and quarterly is how most companies operate. Obviously tech companies are going to push forward ideas and concepts.

LOL melt the ice caps! ?
why not put them next to buildings like hospitals?
you can let them use the heat!
And they PAY you for it!!!!

and schools any big building.
so They dont ADD to the world heat.

No! forget it lets be stupid.

originally posted by: Caver78
Underwater server farms in northern latitudes has a time limited cost effectiveness on financial returns. Meaning it will only work so long as the lower temperatures hold. That may well be for a couple of hundred years, but true corporations plan much further out.

Again there is the issue of dumping server farms on possible archeology sites.

Planning longterm newer networks underwater may be a placeholder, but the reality is we need to design better systems than the fragile spiderweb we all take for granted currently. Spending billions on a system that depends on the current chips an wires makes little financial sense, unless Microsoft an others have tech we're unaware of they're going to roll out.

I gather you're skepticism is a function of concerns of climate change. Even if the surface temps raise 5-10 degrees, there will still be plenty of very cold water at the bottom of the ocean.

originally posted by: buddha
LOL melt the ice caps! ?
why not put them next to buildings like hospitals?
you can let them use the heat!
And they PAY you for it!!!!

and schools any big building.
so They dont ADD to the world heat.

No! forget it lets be stupid.

Depends on where these hospitals you mention are: Milwaukee,maybe, Arizona, not so much.

It's a lot easier said than done for converting excess heat from a server farm into heat/energy for other facilities, and like I said, you still need to worry about the impact on the power grid, and availability of telecom interconnects.

a reply to: SleeperHasAwakened

I dont see why they just dont put servers in huge tanks of Sapphire solution.


www.koetterfire.com...

Ive seen them submerge a computer in this stuff and it did not harm it at all. It can also be cooled and circulated.

originally posted by: yuppa
It can also be cooled and circulated.

You mean use sapphire solution instead of nitrogen? Is nitrogen cheaper?

What I found surprising what the implication that the lack of oxygen reduced failures by 90%. I don't have any reason to doubt the figures for failure reduction, but I'd like to know more about how the lack of oxygen creates that result if that's really the cause of the reduction. They said they pumped Nitrogen into the capsule. I knew heat could age circuits, but I didn't know they were affected by oxygen levels.

a reply to: buddha

Excellent idea's buddha.

originally posted by: AugustusMasonicus

originally posted by: Caver78
That may well be for a couple of hundred years, but true corporations plan much further out.

If you think corporations have a multi-century strategic plan you're kidding yourself.

Cue the inane 'duh Rothschilds' reply.

Yeah, I’ve heard of Japanese companies with ridiculously long business plans, but few U.S. firms look beyond the next quarter. Or so it seems. Elon Musk and Jeff Bezos are certainly looking far ahead. Musk seems to view his fortune as a means to realize his vision. Bezos has so much money he can afford to invest billions without a near-term payoff.

originally posted by: Arbitrageur

originally posted by: yuppa
It can also be cooled and circulated.

You mean use sapphire solution instead of nitrogen? Is nitrogen cheaper?

What I found surprising what the implication that the lack of oxygen reduced failures by 90%. I don't have any reason to doubt the figures for failure reduction, but I'd like to know more about how the lack of oxygen creates that result if that's really the cause of the reduction. They said they pumped Nitrogen into the capsule. I knew heat could age circuits, but I didn't know they were affected by oxygen levels.

Humidity is a big factor in the life and reliability of equipment. Perhaps that's why a nitrogen environment has fewer failures.

Heating is indeed a huge issue with servers.

My first MS gig was os testing in a lab with maybe 200+ PCs running 24/7, in a building with dozens of labs. That building had some of the most serious AC ive ever seen. I had another gig testing multi display in a small office and had maybe 8 high end systems with 5+ video cards each and the AC went out one night and it was 120 in there the next morning. My DEC and SGI systems were the only ones smart enough to auto shut down.

This heat does sound like it could be harvested, but another thing I noticed is servers are almost always backed up by gigantic UPS backups, which are highly hazardous batteries filled with explosive toxic materials. I remember the fire dept explaining this whole thing to me at nvidia when I was again in a giant room full of hot PCs.

I think we ought to focus at least a little more effort on making our tech less toxic and invasive

Doesnt facebook already have a few of these ?

originally posted by: LogicalGraphitti
Humidity is a big factor in the life and reliability of equipment. Perhaps that's why a nitrogen environment has fewer failures.

I'm not sure ... Even with servers running in air, the recommended humidity level is 50% rH. Would the rH recommendation be different in nitrogen instead of air?

If humidity gets too high, corrosion can result. Anything over 70% rH is really bad.
But if humidity gets too low, electrostatic discharge (ESD) can be a problem, so that's why the humidity recommendations are not to just keep humidity as low as possible to avoid corrosion.

However, maybe in an unmanned facility (or test environment), ESD is less of an issue and lower humidities can be tolerated?

These are ASHRAE's humidity guidelines, expanded from the old guidelines of 45%-55% rH (I think 80% is too high, anything over 70% can be a problem if it's sustained, maybe they agree since they still recommend critical alerts if rH gets above 70%):

ASHRAE’s newer 2016 guidelines remain relatively the same with data center humidity, with a recommendation of 50% humidity. Minimum humidity is 20%, while maximum humidity is 80%...

If humidity is too low, the dry air will lead to electrostatic discharge (ESD) which can damage critical server components. Too much humidity will cause condensation, leading to hardware corrosion and equipment failure.

Our recommendations on humidity alerts remain the same as they were in 2005. Early warning thresholds of 40% and 60% relative humidity should trigger alerts. Critical alerts should be sent if relative humidity reaches either 30% or 70%.

originally posted by: Aliquandro
I think we ought to focus at least a little more effort on making our tech less toxic and invasive

I agree. Something seems wrong with these crypto currency generating shops using vast amounts of electricity, probably generated by vast amounts of negative impacts to the environment for something that seems rather ephemeral for all that damage to the environment.

Making Cryptocurrency More Environmentally Sustainable

Last year, blockchain used more power than 159 individual nations including Uruguay, Nigeria, and Ireland. Unsurprisingly, this is creating a huge environmental problem that poses a threat to the Paris climate-change accord.

It’s a brutal, if unintended, consequence for such a promising technology, and “mining” is at the heart of the problem.

That was 2 years ago, it's had 2 years to get worse.

a reply to: SleeperHasAwakened

Just a little. Watched Deadliest Catch an the very real impact of a warming of the Bering Sea came to mind.
But I'm not all crazed over Climate Change in general.

More concerned on how warming up isolated pockets of the northern ocean will impact those smallish areas, how are we gonna keep Trawlers from accidentally ripping up a server farm? Plus we know between Europes coast and the UK there ARE significant archeology sites that have been lost underwater.

Knowing how fragile the whole Internet system already is it would seem stupid to keep investing in more of the same instead of R&D into something better.

Just my 2 cents

a reply to: Caver78

Doubt they'll put them in deep water but just offshore where something can drag them up as and when needed, they still will need power and data so they'll be at depths that a drop ship can operate in so probably less than 100ft deep.

These DC's are cheap to run as its basically just dumping a shipping container into the water and connecting up a few cables and they're expendable in that they'll be full of commodity servers in the 1-2k per server range and over their life span will probably earn 100's of times the actual costs doing azure cloud work.

You also have the advantage of being able to dump them near places where tidal/wind energy are available meaning less reliance on the grid saving more money.

Theres a good reason as well for filling them with nitrogen instead of oxygen as anyone whos seen a server go up and toast a rack will attend to.

originally posted by: Arbitrageur

originally posted by: yuppa
It can also be cooled and circulated.

You mean use sapphire solution instead of nitrogen? Is nitrogen cheaper?

What I found surprising what the implication that the lack of oxygen reduced failures by 90%. I don't have any reason to doubt the figures for failure reduction, but I'd like to know more about how the lack of oxygen creates that result if that's really the cause of the reduction. They said they pumped Nitrogen into the capsule. I knew heat could age circuits, but I didn't know they were affected by oxygen levels.

This was a very salient observation from the article. I noted the use of Nitrogen gas in lieu of regular atmospheric air mix, but didn't give the idea much thought vis a vis failure prevention.

I know that liguid nitrogen is often used as a cooling system in custom PC builds, but not sure what would make Nitrogen a better gas than Oxygen, other than perhaps its dew point/dryness characteristics.

I think Maxatoria is on to something as well, regarding removal of Oxgen to prevent oxidization of components and of course eliminate chance of severe combustion potential.