How Terremark keeps data centers green

MIAMI, FLORIDA - Increasing energy efficiency in data centers is a never-ending challenge. And the task is more complex for providers that have to split their facilities between data centers and colocation facilities, like Terremark, which has facilities around the globe including its largest in Miami, where it's headquartered.

Managing energy efficiency could become even more complicated as a result of government initiatives on the horizon. Ben Stewart, Terremark's senior vice president of facilities engineering, spoke with Connected Planet about his green thumb.

On cooling the colo floor versus the cloud:

We have large colocation floors, so we have a lot of different customers bringing their data centers into our facilities. Most customers only load their servers up to 20% or 30% utilization. But in one part of the floor, we set up our very high-density cloud environment. There all the servers are running at close to 100% utilization, so they're running pretty hot. And we designed the room specifically for that. In most cases we look at close coupled cooling and containing the cold aisle or the hot aisle to make them run as efficiently as possible.

Rather than ramping up all of our air handlers to push more air into an area, we'll drop in small in-line coolers into the cabinet lineup to add more cold air to just that area where we've got the high density. That's close coupled cooling. And where you've really got some high density, you can contain the hot aisle or the cold aisle, preventing the mixing of hot and cold air, which makes it very efficient. You're limiting the cold air you deliver to the servers to that which they need. You can see dramatic savings in energy because you can slow your air handlers down.

(The cloud) is only a small part of our floor right now. Over time, the colocation floor will become smaller and the cloud part of the floor will grow.

On economization:

We shy away from air-side economization - using outside air to cool your facility -because it brings in contaminants. You can filter some of it out, but over time, there's some corrosion associated with it. We prefer the water side, running water through radiators and blowing the cold outside air across those radiators. In the winter, you may never run your chillers. Today, we're seeing fantastic savings (aided by cold temperatures). In the fall and spring, it may be cold enough at night to not run your chillers.

We've installed chill-water storage tanks. At night, we make a little extra chill water, and we fill the tanks with them. That delays the time during the day that we'd have to turn our chillers on. We get a little more efficiency that way. We bank the chill water during the fall and spring months. Of course, you can't do that in Miami. You don't get enough cold air. In Miami we can't do any kind economization.

On power efficiency and fiber diversity: If you can get close to a hydroelectric dam, you're looking at 4 cents per kilowatt-type power rates - very cost-efficient. The problem is, where your hydroelectric dams are, you don't typically have very high fiber densities. It's very difficult to connect to the rest of the world. We're not colocated to any kind of hydroelectric sources because we need to be near where the fiber is. We've been in talks with a delegation from Iceland. They wanted to launch a data center market up there because they can free-cool 365 days a year. However, there's only one major piece of fiber going to Iceland, and if that goes down, you can't use that data center.

On the PUE game:

PUE is a measure of the ratio of the total power coming into a building divided by how much actually gets to the IT equipment. If you've got a building with a PUE of 2, that means half the power coming in is going to the IT equipment, and the other half is going to infrastructure: generators, chillers, air handlers lighting - everything else. The (EPA EnergyStar) rating is going to be based on that. It's going to require data centers to meet some PUE target.

Knowing your PUE is very important because as you make improvements, you can watch that number come down. And you can equate it back to dollar amounts. You can go back to your CFO and say, "Look, I saved you $60,000 a month in utilities. I'd like to get some of that back to reinvest in the infrastructure to save even more money." It turns into a fun game that you can really measure.

When we started measuring ours a little over two years ago, we were at 2.1, in the average range. We're now down to 1.6, which has been an incredible cost savings. We used to just keep the entire floor cold. We'd crank up the air handlers, keep it as cold as we could. We learned we don't have to keep the whole data center cold, just the cold aisle.

Our electric bill is our second biggest expense, salary being the first. Our 750,000-square foot facility here in Miami is only half built, and our monthly electric bill is $660,000. We don't have to get a whole lot more energy efficient to start saving serious dollars.

On emulating Google:

Google is doing very innovative things that have driven their PUE down to a 1.2, 1.14, which is outrageous. One is perfect; that means every watt of energy you're bringing into your building is going directly to your IT equipment, and you're losing none in all your lighting and equipment. They kind of changed the equation dramatically. (But Google) kind of cheats a little bit, putting batteries in their servers; the rest of us really can't do that. They're building data centers inside containers that have a very limited amount of air inside them, which makes it very easy to manage airflow because there's not much of it to manage.

When you're Google and you have a 5,000-server farm, you can do whatever you want because you own that IT infrastructure. We're a colocation facility. We don't own the vast majority of equipment in our facility. We can't dictate to customers: You can only come into our facility if you have batteries in your servers, which no one has.

On the Environmental Protection Agency's rating system: The EPA is launching in April an EnergyStar program for data centers. We're being told if the government is going to outsource IT operations, (the outsourcee) must be EnergyStar-rated. We're keeping a very close eye on that. As soon as it comes out, we'll be applying for it. We do a lot of government business.

On how EnergyStar ratings will affect the market beyond government customers:

We're already hearing it. Right now a lot of data centers are saying, "I've got a PUE of this," "I've got a PUE of that." But they can play with that a little bit. Once you get audited by the EPA and you have that EnergyStar rating, you kind of have a halo over your head.

On cap-and-trade regulation:

That could have a very significant effect on our utility rates. As our utilities here begin buying credit from other areas that have a lot of hydro and nuclear (power), they'll pass that (cost) along to us, and our utility rates will go up. There's still a chance (the legislation) won't pass and we won't have to worry about it.

On blade servers: We aren't seeing a lot of customers using blade servers. They're still in the 1U market. Two or three years ago, when the market was 80 to 120 watts per square foot, they were predicting that by now we'd be seeing 700, 800 watts per square foot deployments. We aren't seeing that at all. It's partly the cost and partly the space.

If you take a legacy data center and you replace all your pizza-box 1U servers and put them into blade servers, all you've done is just taken the power you had in that data center and crammed it into a corner. That makes it very difficult to cool. Now you're out of power, and you can't use the data center space you freed up anyway because you're out of power. The business case for those blade servers made sense in some cases. It doesn't apply industry-wide.

On renewable energy: It's the holy grail — wind turbines, solar energy. Right now it's still too expensive. But the day will come when that becomes economically viable. When you have renewable energy, in theory you're running your data center for free. That's what everybody is driving toward.