Big blackout. What happened? 40

[Skogsgurra]

When I got words about the big outage, I immediatley went to my puter to find out what my engineering friends in the US had to say about it. But no Eng-Tips page available. Of course I can understand that. No power - no Internet.

Power was restored piece by piece and I now find Eng-Tips up and running again. My question is still valid: What happened?

Glad to see you again!

Gunnar Englund, Sweden

 

[jstickley]

skogsgurra --

Do you have any idea the enormity of the system you're proposing? You're looking at needing millions of data points in real-time if you want to monitor every breaker, switch, and line flow in the Eastern Interconnect. I stand by my original viewpoint -- this is probably impossible at this point. Maybe this is not the case for the system you live with in Sweden.

All --

I really think that any talk about "if operators would have just done this, this wouldn't have happened" or "if we had this system to monitor the grid, this wouldn't have happened" is foolishness. Has anyone here besides myself been involved with system operations during a crisis? Operators get inundated with information, especially when there are a number of problems occurring simultaneously. EMS/SCADA systems end up constantly alarming. Scheduling systems have changes that must be implemented. The phone rings off the hook. Supervisors ask questions. It can be a very chaotic situation, given the demands placed on operators in today's industry.

I guess what I'm saying is this, if you must place blame, place it on the adequacy (or inadequacy) of the system design and the way in which the industry expects it to be operated. It's safe to say that the two are quite different -- the system was designed to deliver local generation to local load, but we now operate it to transfer thousands of MW of electricity. To expect it to perform adequately under such high stresses while experiencing multiple outages is unrealistic.

 

[37pw56gy]

Anyone care to speculate on the possibility that all this has something to do with automatic versus manual control of phase angle regulators scattered throughout the system?

 

[SidiropoulosM]

37pw56gy, if the phase shifter automatic controls are not disabled following a major contingency, it is possible that the phase shifter controls could aggravate the situation.

I'm not betting the farm on this one yet, but you've made a good point.

 

[piterpol]

Due to the importance of this thread I have been printing it as experts are expressing their point of view day after day. But today I found this page quite widely expanded so I cannot print it on a letter size paper. Please tell me some of you if it is possible to narrow it to a letter size width and how can I do this!
Thanks a lot.

 

[SidiropoulosM]

Piterpol, at the top of this page you'll find a little window called "e-mail it". Click on it and send the thread to your own email. You can then print the email and it will print fine.

 

[jbartos]

Suggestion: Normal generator protection devices, which can trip on the grid unusual conditions, are:
1. Undervoltage. This can cause the Generator Automatic Voltage Regulator (AVR) to be out of its voltage regulating region
2. Over/under Frequency Protection Relay. This relay protects the generator from damage if the relay setpoint, upper or lower, is passed.
3. Some generators can trip on "no load", i.e. sudden loss of 100% load. Others can withstand it. It just depends on the genset design.
4. Periodically Cycling Load. That implies some oscillations in the grid, which can potentially cause electromechanical resonance and subsequent generator trip due to oscillations.
5. Etc.

 

[SidiropoulosM]

Jbartos, great info. Since most of the generators tripped were nuclear, can you point out which of these protections apply to nuclear generators?

Also: are there any other protections that are specific to nuclear generators?

 

[jbartos]

Suggestion to the previous posting. The nuclear power plant generators are relatively large in MVA or MW, e.g. over 300MW and up to about 1200 MW. They are usually sensitively protected since they are expensive. Therefore, they trip first. Thanks goodness that they have tripped. Now, we have power again from undamaged gensets. Incidentally, some nuclear power plants offered tours to the public in the past. One might have seen the nuclear power plant turbo generator very closely, e.g. on the roof of the turbine building (Salem, NJ).

 

[margaret695]

skogsgurra: I can envision that the system that you are thinking about would be a very useful tool. But the thought of taking all the as-built drawings for the existing transmission lines, as well as all the revisions and modification, length of lines, types of cables, protection devices, doing surveys, etc., etc, etc, and inputting said data accurately into a program sort of boggles my mind.

But I am wondering if the notion of having it part of the specifications for new transmission lines, that the contractor building said transmission lines... or modifying said transmission lines... would have to submit as part of his contract accurate data base lists... prepared in a strictly regulated manner.... tied to autocad drawings.... of all the material and equipment used. That way the data could be compiled slowly... and hopefully accurately... over a number of years. Okay... maybe decades.

Any ways... if the data was compiled by those actually working on the construction and maintenance of the transmission lines... that would decrease the labour costs for the data input of the system that you are proposing. Especially for new transmission lines, as the lists for cost estimates possibly could be tweaked into an appropriate format.

I do agree that methodologies for presenting technical information to non technical bean counter types is very very important.

Maybe some one could write a movie script. Hey.... a lot of non technical types got the safety related message behind "the China Syndrome."

Or possibly we could get the video game people involved.

Please note that I am only partially joking.

Margaret

 

[margaret695]

fundam: I will admit that I checked out the records for sun spot and solar wind activity a couple of days back.

Didn't look like anything unusual was happening on the 14th.

Also, by my understanding, solar activity is more likely to affect transmission lines in the far north... like the ones from James Bay, etc.... and I haven't heard of any problems on those lines.... and my lights didn't go out. Also didn't hear of any problems with the communication satellites, which seem to be more sensitive to sun spot activity then transmission lines.

Margaret

 

[margaret695]

have been sitting here pondering the subjects of power factor and harmonics, etc.

The types of loads in buildings has really changed over the last few decades. We got lots and lots of computers. Offices used to run about 2w/sqft and now they often run at about 4w/sqft, or more.... because of all the computers... with their switching power supplies. Then there is all those electronic ballasts out there for fluorescent lights and HID lights, along with the usual induction type loads like motors. And motors have variable speed drives often nowadays. And electric heating is often controlled by triacs, etc., etc., etc.

Any ways.... where I am located, the electrical utility bills on either the demand, or 90% of the kva, so as long as the power factor is above 90%, whomever is paying the utility bills does care... and even if the power factor is running just under 90%, my clients will ask for the pay back period of installing capacitor banks before actually installing capacity banks.

Makes me wonder if the power factor for the over all grid perhaps has creeped upwards during the last decade. Also makes me wonder about the levels of harmonics. As from what I have seen, the voltage data available for this event is rms.

Hmmmmmmm...... of course I could be way way off base.... but does make me wonder.

Margaret

 

[margaret695]

Okay... so I am typing impaired....

Meant to say .... so as long as the power factor is above 90%, whomever is paying the utility bills does NOT care.

Margaret

 

[margaret695]

Going to sleep now... meant to say that the power factor could be creeping downwards.

Margaret

 

[RAMConsult]

There are many other layers of generator protection that can be applied at the generator terminals. These include Overvoltage, Phase current/voltage imbalance, Phase Reversal and High Negative Sequence Current. In addition, there are control systems (fuel, turbine and generator) that are less obvious but in this scenario may prove to be equally important to understanding what happened.

The generator has an Automatic Voltage Regulator whose primary function is to control the field current so that the terminal voltage is held to a relatively constant value, but there are usually other functions that it provides including Line Drop Compensation, Reactive (VAR) current control/limiting, Field Over/undercurrent and System stability control.

Note that all of these are control functions that operate relatively independently of the protective relaying functions even though they all use the same control variables (current and voltage) for inputs.

It is also important to consider the Turbine Control System (Governor)which manages the raw mechanical power, speed control (normal, over, under) and stability of the turbine. It is the interaction of these diverse subsystems that have an integrated impact on the overall functioning of the entire grid.

 

[jstickley]

piterpol --

A bit of advice -- take the "words of wisdom" you find on this site with much caution. This site, as wonderful and generally informative as it is, has no screening process to ensure the "experts" who post here really know what they're talking about.

In this thread alone I've seen a lot of ideas that are pretty far out there in terms of the reality of day-to-day power systems operation.

all --

It was stated that that most of the generators that tripped in this event were nuclear. This is not the case. Hundreds of units tripped -- the problem was not specific as to the type of plant it affected.

Also, note that often the reason generators trip is not due to the generator relaying designed to protect against faults, but rather interal plant loads that are necessary to keep the plant running. We've had several instances of voltage fluctuations on the grid causing plant auxiliary loads to trip (variable speed drives in particular seemed to be particularly sensitive).

 

[SidiropoulosM]

Jstickley, if someone said that "most generators that tripped were nuclear" he probably meant "many" instead of "most". There are many contributors here for whom English is not their first language. For some of them the words "many" and "most" may be interchangeable. In any event, it's not a big deal.

What I find amazing is that most contributors here will show a great deal of tolerance and will go out of their way to understand a question, even if written in bad English. I think this attitude enriches the forum and this is the type of attitude we want.

 

[jstickley]

SidiropoulosM --

Don't take this personally, but clarity IS important. As I noted in a previous post, many of those here reading this thread are not informed about the industry, and my clarification was intended to prevent confusion amongst those readers rather than to nitpick those whose first language may not be English.

To answer your question regarding generator protection, I would strongly suspect that the relaying schemes that protect nuclear generators are similar to those that protect other generation sources. I can't imagine that the source makes much difference in the type of relaying required to protect the generator from electrical faults (jbartos's point is excellent, but I think it rings true with all types of units -- larger units will likely have more in-depth protection schemes).

The difference I see between varying sources of electricity (nuclear, coal, combined cycle natural gas, simple cycle natural gas, wind, solar, etc.) that could affect tripping characteristics is in the plant itself -- the motors, pumps, and equipment that make up the plant and the varying processes by which the sources convert prime movers to electricity. Due to differences in the equipment used, plant tripping characteristics likely vary among different plants. I would suspect that nuclear plants have many redundant safety systems, and these systems may be more or less tolerant of system dynamics (I can't say which, as I don't have direct experience with any nuclear plants).

 

[Skogsgurra]

jstickley,

I will make myself more clear. The NERC has been active for almost forty years. Their main task is to find means to avoid big outages like the one we are talking about.

In my opinion, they have not been sucessful. The two hours before the total collapse could have been used in a much more efficient way. Even a modest observer - a lot simpler than the one I envisage - would probably have helped.

What I want to say is not that a complex system should be built right away, but that one should consider and think about such a system during the next ten or twenty years. And, frankly, I cannot understand what you have against that.

This is my last posting with regards to the observer/estimator. There are a lot of other things to discuss. Let's do just that!

Gunnar Englund

 

[jstickley]

You guys are really getting me going today....

skogsgurra --

First off, I have no problem whatsoever with your idea -- it's just that on the scale mentioned, implied, or maybe just what I interpreted, it's not feasible. Where did you get the idea I'm against such a thing? I've stated that I believe that every utility should have a state estimator that can perform security analysis.

Regarding NERC, are you blaming them for the outage? Last time I checked, NERC does not have system operators sitting there monitoring the system. NERC's purpose is to ensure the reliability of the system, true, but it does so through establishing reliability standards, monitoring compliance with those standards, facilitating education of industry personnel, coordinating assessments of the interconnected system, etc. NERC does not have direct authority over any individual utility regarding day-to-day system operations. Here's a link with information about NERC:

http://www.nerc.com/about/.

I believe that as more information becomes available, you'll find two things -- the system was planned according to industry-accepted reliability criteria, and that system operators did everything they could to prevent this outage from happening.

If the system is found to be planned inadequately, that's one thing. However, if it's planned according to established and accepted criteria, you've got to answer the following: is our current criteria adequate, and if not, what is?

If the cascade is found to be due to the outages that occurred from 14:06 to 15:06 CST (this if from NERC's preliminary report,

http://www.nerc.com/pub_doc/PreliminaryDisturbanceReport.pdf),

I believe that investigations will show that operators were doing everything they could to get those lines back in service to prevent further problems. The fact is that when outages occur, often times it takes time to get them back in service. Even if the line is physically intact, relays lock out, and sometimes have to be manually reset by sending people to the stations. If it turns out there was something that could have been done aside from putting the lines back in service, don't forget that hindsight is always 20/20. I guess the tough question regarding system operation is this: were operators taking whatever actions were reasonable and prudent to prevent this from occuring? My gut feeling is that this is the case.

Anyway, I've rambled on enough. I believe the bottom line in this whole event to be this: STUFF HAPPENS (I know, the phrase is different, but profanity is not acceptable here). Sometimes, in spite of all things, we just can't prevent problems from happening.

 

[Siddhuca]

I have been reading all the comments in this forum. I
find it extremely unbelievable that when the same 345kv line tripped twice , it was reclosed immediately without analysing the cause of the fault.
Having two hours after the first trip "with good communication and instructions from a regulating command this blackout could have been avoided.
Islanding should have worked.