Las Vegas / Bellagio Hotel Power Failure 4

[jraef]

For anyone interested, here is the inside scoop on what happened at the Bellagio 2 weeks ago. The media stories avoided most of the technical issues of course, but my curiosity got the best of me. I couldn't understand how a cable failure could take out the backup power systems as well. It turned out to be an interesting lesson in being "penny-wise but pound-foolish"!

They have (had?) a 2000A 15kV main switch from the utility feed that had a cable connection to a lineup of 100A distribution switches. Each distribution switch fed a 12.47kV-480V substation for various load zones. Each load zone had a 12.47kV geneset tapped in with a "T" connection to the primary of each sub.

The original fault was in one cable from the 2000A main to the main bus of the distribution switch lineup. It was a low level insulation fault that apparently arced and smoldered for some time before being cleared.

The major catastrophy was that in an effort that saved $2million in cable costs by going with 12.47kV gensets and T connections, they unfortunately stuffed all of these 15kV cables into the same duct bank, along with the feeder that failed, and of course it was on the bottom of the duct allowing the damage to spread upwards. So the relatively low cost failure of that one cable took out not only the entire
distribution system but also all of the backup systems as well!

The rumored total is that the outage has cost them $20 million so far.

You didn't hear it from me though ;-p

"Venditori de oleum-vipera non vigere excordis populi"

 

[edison123]

jraef,

After all, it is las vegas. they probably thought the dice were loaded in their favor.

 

[RonShap]

Your saying that they had a 15kV 2MW genset tapped into each 100A utility distribution feeder without a transfer switch and the feeders out of these gensets were routed with the 2000A service conductors?
The physical locations don't seem to make sense.

 

[alehman]

It sounds as if the generators were connected directly to the primary at each USS. Presumably there is a disconnect somewhere between the T tap and the generator. Perhaps there was no disconnect between the T tap and the faulted duct bank, which would have prevented the generators from operating. In that case one extra switch per generator might have saved the day.

I was recently at the the Bellagio. It is quite impressive. Most smart gamblers know you can loose even with loaded dice.

 

[jbartos]

Comment: It is not unusual for a large hotel casino to have four electrical utility services. Therefore, these have never been down yet with such a robust redundancy.

 

[alehman]

If many cables were in one duct as it seems, the initial failure may have been a result of overheating.

 

[jraef]

Yes, they do think now that the duct was overfilled and that caused the initial failure, although it is still "under investigation" (meaning they are probably desperately seeking someone to take the fall). The generators did operate, but the cables had faulted, ALL of them, so they generator breakers immediately tripped off-line as well. They had presumably done it this way to iliminate the transfer switches. I have not personally seen the gear so I can't speak from direct knowledge. I got this from an engineer who was called in to help supervise replacing cables. They hired just about every medium voltage qualified splicer in Nevada and got it back on-line in 3 days.

Then they probably called every slot machine technician and tweaked them all a smidgen to pay for it all!

"Venditori de oleum-vipera non vigere excordis populi"

 

[advidana]

I am in Vegas. There is a lot of stories out there about what happened at the Bellagio. I hear a different story every day. I would wait about a year before the truth is known if ever.

 

[jbartos]

Comment: Proper physical separations of various parts of redundant electrical power distribution systems and necessary barriers are frequently seen unimplemented, except in the Nuclear Power Plant after TMI accident.

 

[DanDel]

This is one reason why some states (MA in particular) require fire-rated isolation between normal and emergency systems.

 

[dpc]

You have to distinguish between "emergency" and "standby" power systems. Although the terms are sometimes used interchangeably, the NEC has very different definitions and requirements. Emergency power systems are those considered essential for safety to human life. For these systems, the NEC does require separate raceway systems be used, or use of a sprinkler system.

So it would depend on whether the local AHJ decided this was an "emergency" or a legally-required standby system.

My experience in Las Vegas has been that these huge casinos are so unique that they don't fall neatly into typically building categories and fire protection/life safety requirements are worked out through negotiations with local inspectors.

 

[jbartos]

Comment: Redundant services are supposed to be a good solution since there are more power supplies available than covered by a minimal emergency or standby power system only.

 

[busbar]

Off topic a little, but doesn’t one of the major clubs there have a switchyard with a 115- or 138-kV loop and a pair of ~60MVA transformers?

 

[alehman]

Emergency distribution circuits as defined in the NEC must be in separate raceways except at utilization equipment. In buildings over 75ft high or >1000 occupants that are not sprinklered, there must be a 1-hour or equivelant fire separpation for emergency feeder circuits.

 

[jraef]

As I understand it, these were backup power systems, not emergency power. The emergency power systems apparently worked as far as exit lighting etc., so probably fire sprinklers were available too (luckily they didn't need to find out). The bean counters will of course call this an emergency, $20 million is a lot of money to loose in 3 days, even for them! Data loss on their bookings and reservation computer systems continues to haunt them as well. Several conventions were cancelled or moved as a result so the losses continue.

"Venditori de oleum-vipera non vigere excordis populi"

 

[jbartos]

Comment: It appears that a good solution should include more electrical services not necessarily rated on the full load.