Grounding in Data Centers

[Dumbo2929]

I am looking for people's experience with data centers, in regards to the grounding of rack enclosure cabinets. I have a customer you has lost several power supplies. The utility service ground is fine (less than 1 ohm), as is the panelboard grounds for the circuits feeding the racks sitting on the raised floor. There is an under-floor ground grid, but none of the enclosure cabinets in the data center have their frames grounded. They appear to be relying solely on the plug ground of the 120V cord power strips. Any thoughts on what is typically done with rack enclosures and grounding?? Any obscure code or IEEE requirements anyone is aware of??

 

[alehman]

Common practice is to bond the cabinets to the floor grid, although reality is it frequently doesn't happen.

Have the power supply failures been related to any other events such as thunderstorms, transfers to generator, etc?

You might take a look at the IEEE Emerald Book.

 

[ScottyUK]

I assume that the usual cause of dead power supplies - heat - has been ruled out? Room ambient isn't much use for measuring hotspot temperatures in modern server racks. At a previous employer we used a thermal imaging camera back when they were really expensive toys. These days they are fairly cheap.

Regarding rack earthing, over here in the UK the Regulations are the minimum required for safety, not the design requirement for reliability and longevity of equipment. I'd expect to see the cabinet frame earth bonded down to the main earth grid.

Slight aside: when we have traditionally installed racks we have connected them to one or more BS4343 sockets installed under the computer room floor. We consider the rack as a piece of 'portable' equipment by virtue of being connected to a socket and size protection to meet the 0.4s fault clearance time required by the UK regs. Fixed equipment has a 5.0s fault clearance time although obviously we don't design right up to these limits. I'm curious whether others treat banks of racks as being 'fixed' or 'portable' equipment and whether you hardwire them or use a plug & socket?


----------------------------------

Sometimes I only open my mouth to swap feet...

 

[alehman]

Scotty,
Good point about temperature. My experience has been that there is a wide range in quality of power supplies. They are probably the most common failure item in server equipment.

In the U.S. we almost always connect racks with cords and plugs. Overcurrent protection requirements in the U.S. are the same either way.

 

[EEJaime]

We bond the racks to the grounding grid along with the raised flooring grid and any other metallic surfaces. We generally like to use overhead power drops alongside the wireways and keep the below raised floor area for data cableing as much as possible. Since in my area, (Southern California, USA), we have a large amount of seismic anchorage required in many installations, the area below the floor becomes quite congested.

I would certainly look for a heat source or other cause for the power supply failures, it seems a stretch to think that an ungrounded rack would directly contribute to a power supplies' failure. If the power supply is properly mounted and connected there would not be much of a connection between the racks and the power supply.

Is the room properly air conditioned? Is the air supplied from the raised floor? Is there a supply air opening below the racks?

Does your system utilize UPS equipment?

 

[dpc]

This might give you something:

http://www.erico.com/products/SRG.asp

 

[wirenut60]

If your data center is built per NFPA 75 and the electrical installed according to Art 645 in the NEC (NFPA 70), then the frames of the racks are required by code to be grounded separately from the cord and plug. Typical grounding arrangements would be a bare 2/0 grounding conductor run under the raised floor and bonded to the raised floor support systems. (Some raised floor manufacturers state that their system is UL listed to provide a solid ground thus making the separe bare conductor unnecessary). From this ground, jumpers are installed to the equipment frames. I'm not sure that the failed power supplies are a result of lack of a frame ground, since the equipment is also designed to be installed in areas where the frame grounds are not required. A few years ago, i did see where an improperly isloated ground circuit caused power supply failure. The IG wire was connected to a separate driven ground that was not bonded to the system ground. Hope this helps...

 

[ScottyUK]

Have you checked power quality? One of the most peculiar failures I'd seen (at the time) occurred when active power factor correction was a new idea. Our IT guys had installed something - a great big server IIRC - and plugged it in to our UPS. The UPS output waveform was not especially good and it got worse when this item was plugged in. The failure occured because the active PFC front end was trying to draw a current which matched the waveform of the voltage input, complete with a multiple zero crossing and high magnitude low order harmonics. The circuit did a fair job of tracking this messy supply voltage until the magic smoke got out. Nowadays those active PFC units are much more common - might be worth looking at the supply quality and the type of technology used by the power supply.


----------------------------------

Sometimes I only open my mouth to swap feet...

 

[Dumbo2929]

We are in the process of a power quality survey, so more to follow. I appreciate everyone's input; excellent as usual.

 

[rob46]

1. Per Article G.5.1.6 of ANSI/TIA-942-2005 "Telecommunications Infrastructure Standard for Data Centers" bonding to the building grounding system shall be provided by installation of "#6 AWG or larger conductor to each computer or telecommunication cabinet, rack, or frame. Do not bond racks, cabinets and frames serially".

2. See also IEEE sTANDARD 1100.

 

[BJC]

Google up TM 5-690. It has a good section on grounding and it's free.

 

[peebee]

The rack grounding would have little to do with your power supply failures. So long as you have good grounding to the equipment via the power plugs, you should be able to rule out grounding as a source of your power supply problems.

 

[rbulsara]

I am with peebee. Do not get started on a wild goose chase just because server people are looking for an excuse. Focus on finding real cause, most likely what ScottyUK suggested.

Next in line (probable causes) would be the quality of the power supplies themselves. They could be just dirt cheap items. The look for look for votlage transients.

 

[rbulsara]

(Correction: The last line in my previous post should have read: Then look for voltage transients, etc.)

One of the probable causes: Check the voltage rating on the power supplies vs actual voltage. Are they really rated for your voltage? Specially the server/computers are not made in the country they are used in.

For example, make sure they are not rated for 110V and your actual voltage is 120 nominal but is 125V actual?

 

[peebee]

Rack grounding and underfloor grounding are there to help protect signal integrity, that is, to provide a good high-frequency ground. It may be worth exploring, but not for the stated reason (power supply failures). In my mind, it's over-rated, and not nearly as important as it used to be when signals were primarily unbalanced and referenced to ground (vs. more modern balanced, unreferenced signals).

The PC in my office, and the 2 I have at home, all suffer the extreme indignity of being forced to chug away day and night without benefit of any signal reference grounding system. Perhaps my data thruput suffers as a result. But the power supplies hold in there just fine, despite being some of the cheapest PC equipment available anywhere.

 

[peebee]

PS - regarding TIA 942 - I can state from first-hand knowledge of the TIA 942 authoring process that the only reason SRG's were included is because lots of people like them and "we've always done it that way". It pulls heavily from FIPS-94, which is also a nice & helpful document but is long outdated (and out of print). TIA 942 was negotiated, not engineered, and the SRG grid spacings stated in there were similarly driven much more by politics than science.

TIA 942 is a great first-pass attempt to bring some standardization to the data center design process, and to provide a good starting-point reference and list of typical considerations. It's a great overview for architects and engineers who have never designed a data center before, and for the data guys. But it is only that, and should not be misinterpreted as anything nearly so stringent as a true code or building standard.

 

[pqsl]

Did you have a batch of these power supplies failing during short period of time and then failures stopped?
Did you do any investigations on Power supplies?
Any surge suppressors on the 120/208V?
Any UPS power supplies?
Our client had this happening 2 years ago.
10 PS in less than 1 month.
Power Quality survey did not find anything.
Investigation of some brand name PS revealed design flaw in some PS on component level.
Good luck please post results.
Actually I am surprised that this type of problem does not happen more often. Maybe IT people just live with it.