Big blackout. What happened? 40

[Skogsgurra]

When I got words about the big outage, I immediatley went to my puter to find out what my engineering friends in the US had to say about it. But no Eng-Tips page available. Of course I can understand that. No power - no Internet.

Power was restored piece by piece and I now find Eng-Tips up and running again. My question is still valid: What happened?

Glad to see you again!

Gunnar Englund, Sweden

 

[jstickley]

davefitz --

Let me give your questions a shot.

1. No, and maybe. The distribution systems had nothing to do with the blackout -- bulk transmission problems triggered the event. Doesn't matter how great your distribution system is if you can't get the power to it. FACTS devices MAY have been helpful, but hard to tell without full details of root causes of the event.

2. Maybe. Restoration of islanded systems is a tricky procedure. I would bet that even if some generation were rapidly restored, it would have still taken a couple days to completely reconnect the system.

3. What design margins are being reduced due to solid-state components? Guess I need more clarification before tackling that one.

4. To my knowledge, lines are not licensed to carry certain amounts of load at all. They may be designed to certain criteria (sag, tension, clearances, etc.), but the rating of the line is dynamic in nature due to the variety of factors that influence its current-carrying capability (say that five times real fast), assuming of course that the line is thermally limited. Yes, we do tend to push the envelope more now on conductor ratings, but the system is also stressed in ways never anticipated when designed, and system operators and engineers are forced to find ways to transmit more energy over the same facilities. That being said, I do not believe any self-respecting engineer or operator turns a blind eye toward facility ratings. This operational necessity is a symptom of a larger problem -- the difficulty in expanding the system to meet the extreme demands of the electricity market (which has been addressed in many ways in this thread already).

Now for a little rant:

I don't know about others here, but I'm REALLY getting tired of reading articles and interviews where people are using this whole event as fuel for their own agendas. I'd really like to just see the investigation run its course, then have conclusions made at that point (and blame placed, if appropriate). No offense intended to those that have placed links to such.

Anyway, let me reiterate a point I've made before: no power system is 100% reliable. The question to be answered is this: is the system planned and operated in accordance with established reliability criteria? If so, the question becomes: is the reliability criteria we have good enough?

 

[lengould]

Came accross this Sept 5 interview of FirstEnergy CEO. Guy's good, you come out of the read feeling sorry for them. Provides very few real facts except "only 40% load on tripped line", "long record of reliable operation" of lines involved.

Definitely a friendly treatment by (Ohio based) reporter. Don't know.

http://www.ohio.com/mld/beaconjournal/6704181.htm

 

[lengould]

Also this from Sept 5, 4:57

http://asia.reuters.com/newsArticle.jhtml?type=topNews&storyID=3392866

quote<A transcript of telephone calls between FirstEnergy and the Midwest power grid operator showed chaos and confusion in FirstEnergy's control room in the hours before the blackout.

&quot;We have no clue. Our computer is giving us fits, too. We don't even know the status of some of the stuff around us,&quot; an operator at Akron-based FirstEnergy said in the transcript.>

Then further:

quote<Another member of the task force said it could take up to a year to complete a report on the cause of the outage and recommendations to prevent another one.>

They won't talk until a year? Something smells bad to me.

 

[spitfire]

Is this report of interest to the current discussion? [I think it might be, but I am just getting up to speed here.]

http://www.selinc.com/techpprs/6141.pdf

It is about a Mexican blackout in February 2002 that was monitored and documented and evidently included effects that were novel.

 

[jstickley]

lengould --

Something may &quot;smell bad&quot; to you, but I'd imagine their estimate on time frame for the report is about right. Power systems are quite complex, and it will take a lot of time just to sort through the MASSIVE amounts of data that have been requested for the investigation (I can't find the letter online, but trust me, it's a LOT of data), let alone determine every detail that contributed to the event.

Granted, it may be somewhat clear what the root cause was just from initial, high-level data (time frames of line outages and generator outages, for example), but without digging into the details, we'll not truly understand everything that contributed to the event. If we don't get a full understanding of all of those contributing factors, it would be easy for knee-jerk reactions to dictate industry change, possibly causing bad policies and/or legislation to be enacted.

 

[SidiropoulosM]

lengould, it does smell a bit. One year is far too long. There's no reason whatsoever why data collection, complete analysis and report should take more than three months. One year is ridiculous.

spitfire, the report you've posted is quite relevant and touches many of the problems that arise from running the system close to stability limits.

 

[cuky2000]

Still I am curious to know what happened

The following are few expressions from the release transcripts of telephone communications provided by the Midwest Independent Transmission System Operator, including exchanges between MISO grid monitors and FirstEnergy, during the afternoon of Aug. 14.

1- &quot;We have no clue,&quot; ..an engineer at the First Energy Corp.
2- &quot;Something strange is happening,&quot; an MISO technician, Don Hunter, told the Ohio utility, not sure what was amiss.
3- &quot;I've got to find my calculator,&quot;….. he said, trying to get a handle on the power fluctuations. ( I am curious what could be done with a calculator during this event)
4- &quot;We've got something going on,&quot;…..for the next 20 minutes there was confusion over what lines were out and what the implication might be for the power grid.
5- &quot;We have no idea what happened,&quot; ….Snickey said. &quot;We have no clue. Our computer is giving us fits too. We don't even know the status of some of the stuff (power fluctuations) around us.&quot;
6- &quot;I called you guys like 10 minutes ago, and I thought you were figuring out what was going on there,&quot;
7- &quot;Well, we're trying to,&quot; Snickey replied.
8- &quot;Our computer is not happy. It's not cooperating either.&quot;
9- &quot;I can't get a big picture of what's going on,&quot; Hunter fretted.
10 - &quot;We won't jump to conclusions,&quot;… Energy Secretary Spencer Abraham said.

 

[lengould]

star to cuky2000: item 10, good bit of humour.

 

[lengould]

Just studying NERC projections for all regions on North America 2000 to 2010. Interesting.

http://pro.energycentral.com/professional/industrydata/loadgrowth/default.cfm

MAIN, which is the area whicn blacked out, claims to have existing 64,170 MW capability 2001, and 74,101 summer demand forecast for 2010, an increase of 9,931 MW predicted to come mostly as dual-fuel turbines (4,677) and combined cycle Nat. Gas. (768) plus out-of-region purchases (2,814).

SERC (southeast except florida) by comparison is huge, 156,058 2001; 192,306 2010; increase 36,250 all gas turbine or combined cycle.

WSCC (most of US and Canada west coast) is also huge, at 130,892 2000; 197,741 2010; increase 66,849 again all (95%) gas turbine or combined cycle gas.

from WSCC &quot;Adequate supplies of natural gas are expected to remain available during the next ten years. However, as demonstrated by California’s experience in 2000, natural gas-fired generation may be subject to uncertainties during periods of unstable natural gas prices.&quot;

SERC is planning to add 2,600 miles of transmission in that time, mostly 230KV (%not avail), and WSCC is planning to add 3,000 miles of transmission 230KV or above. (+4.4%) MAIN data not available, though I have seen it elsewhere, and if I remember, the percentage was smaller.

Other than the fuel choices for expansions, these analyses looked rational and done by knowlegeable persons. We should remember that the types of generation planned as addition don't tax transmission the same as big hydro, or even central thermal and nuclear.

Point is, I fail to see where US$100 billion might be spent usefully on transmission. Figure 10,000 km of 2 Gw lines in 300Km pieces = US$10.8 billion would likely add some significant stability if reasonably placed. I still have a lot of difficulty with that politician's 100 billion figure, and am worried about the interests involved. How did he come up with that number? 10,000 km circles continent 2 or 3 times, 2 Gw is a measurable percentage of entire MAIN generation forecast for 2010.

ref &quot;ABB - for DC xmission + stations 2 Gw 300Km in $US = $325,000,000&quot;

I am also worried about plans to expand all future generation (e.g. 95%), with Natural Gas fired generation with no debate on how rational that is.

BP's world energy forecasts indicate N Gas proved reserves in trillion cuM.
1982 1992 2002 2003
USA 5.78 4.73 5.02 5.19
CANADA 2.75, 2.71, 1.69, 1.70
MEXICO 2.15 2.01 0.84 0.25
TOTAL N Amer 10.67 9.45 7.55 7.15

Consumption in billion cuM.
1982 1992 2002 2003
USA 514.0 583.4 667.5 na
CANADA 53.7 71.7 80.7 na
MEXICO 29.0 28.9 42.1 na
TOTAL N Amer 596.7 684.0 790.3 na

Easy to see that as of 2002, proved reserves only come to 9.0 years consumption. I know, proved reserves aren't all there is, but still i've seen realistic estimates of serious shortages by 2030.
That trend is going the wrong direction for my taste, given winters in canada.

I know about LNG ships, huge reserves in Russia but still, shouldn't new generation use a different fuel? Even oil, so much cheaper to transport by ship. 300 yr coal reserves. I'd vote for rationally installed nd operated nukes. Sure wind isn't going to do it, just a subsidy grab all it is so far. Oceans? Not even tested yet, wait 'till enviros get a look at it. Solar so far costs more energy to manufacture than the cells ever generate.

Also seen several items make me suspicious about accuracy of &quot;proved reserve&quot; figures.

&quot;In the late 1980s, OPEC countries added as much as 300 Gb to their reported “proved” reserves although only about 10 Gb were added from new discoveries.&quot;

http://www.hubertpeak.com

This should probably be different thread, but it's late.

 

[lengould]

For anyone still following, this was from Reuters Sept 4, published on Energy Central Professional.


<quote>
&quot;U.S. infrastructure needs seen at $1.6 trlllion&quot;

&quot;The American Society of Civil Engineers said the total five-year cost of the work needed on items from roads to drinking water systems and schools has surged to $1.6 trillion from the $1.3 trillion it estimated two years ago.

&quot;The lights go out on Broadway. What can happen if America fails to invest in its infrastructure? Anything,&quot; the group warned in a statement.

Electricity industry experts blamed aging infrastructure for last month's cascading blackout in cities across the Midwest and Northeast United States and the Canadian province of Ontario<quote>

Also carries this

<quote>
Public power companies served notice Monday that they are unlikely to sign off on proposals that would allow the operators of the electrical-grid system to increase their transmission charges to help finance infrastructure improvements.

The rate that the grids charge now will be adequate if the money is properly reinvested into the infrastructure, officials said.

The Federal Energy Regulatory Commission &quot;allows a 12 percent rate of return on transmission charges, and that is quite adequate,&quot; said Alan Richardson, president of the American Public Power Association
<quote>

Also carries a discussion of heated debates among engineers at this weeks IEEE conference.

By Dan Piller, Fort Worth Star-Telegram, Texas -- Sept. 9
<quote>
The wrangling over who -- and what -- is responsible for the blackout dominated discussions among electrical engineers at the annual IEEE Transmission and Distribution conference at the Dallas Convention Center
<quote>

Guess i'm not the only one.

 

[lengould]

from

http://pro.energycentral.com/professional/news/power/news_article.cfm?id=4122003

article Sept. 8 (Utility Spotlight)

<quote>Joe F. Colvin, president & CEO of the Nuclear Energy Institute, pressed in his response to the Committee for a &quot;diverse portfolio of fuels and technologies&quot; to provide flexibility and the ability to &quot;balance economic and environmental considerations.&quot;

&quot;Looking forward, we believe that lack of investment in our nation's critical energy and electric power infrastructure is a major problem,&quot; he added. &quot;We are not investing enough in new base-load coal and nuclear plants and we are not investing enough in new electricity transmission.&quot;

He pointed to an EEI assessment showing that about 183,000 Mw of electric generating capacity in the U.S. is 30- to 40 years old and approximately 104,000 Mw is 40- to 50 years old. &quot;That represents 35 percent of our 800,000 Mw of installed capacity, and is clear evidence that we are underinvesting - relying too much on old, less efficient, more polluting generating capacity and not investing in new, more efficient and cleaner facilities.&quot; In the area of electricity transmission, he added, investment has fallen by $115 million per year for the last 25 years, and transmission investment in 1999 was less than one-half of the level 20 years earlier &quot;despite dramatic increases in the volumes of electricity being moved to market.&quot;

He cited another EEI study showing that simply maintaining transmission adequacy at is current levels would require a capital investment of $56 billion by 2010, equal to the book value of the existing transmission system.
<quote>

Somehow I get nervous with numbers like $56 billion. Is that justified?

 

[spitfire]

I stumbled across a paper by B.A. Carreras of Oak Ridge National Labs that was presented in January 2003 at an IEEE conference in Hawaii (International Conference on System Science).

http://www.computer.org/proceedings/hicss/1874/track2/187420065babs.htm

It speaks to probabilities of large blackouts due to efforts at mitigation. I.e., &quot;apparently sensible efforts to reduce the risk of smaller blackouts can sometimes increase the risk of larger blackouts.&quot;

 

[busbar]

It seems authors of the IEEE-ICSS paper might have since felt serendipitous and sickened at the same time, given recent events.

 

[etronics]

Hi all,

I have enjoyed reading all the opinions and theories but unfortunately most of them miss the mark.

It is quite clear that the Midwest ISO did not have a properly working information system during the event. This lead to no response by the ISO to mitigate the outage. Why there was a computer system problem needs to be investigated and why there wasn't enough redundancy built into the system is also an issue that needs some investigation. Having said that, I believe that the black out could have been mitigated if the Midwest ISO operators had good information to work with. Why other neighboring ISOs did not respond when they had working computer systems is another issue. They may be more resposible for the event being as far reaching as it was. Lets not forget the many ISOs that did respond correctly to the system instablilty and isolated themselves as they should have.

Another point is that was never mentioned is the fact that every day the grip operates correctly, isolating problems all over the country so that events like this are not a common occurance. Unfortunately when the grid operates as designed, its not front page news. I don't think we need to panic over this blackout but there are many improvements that do need to be made, and I'm sure we will still have another blackout sometime in the future. Just like cars that will always breakdown so will the electrical system. We can only minimize the problem, we will never eliminate it no matter what we do. A perfect system by imperfect beings is impossible.

Lastly, even though others mentioned this, I found it remarkable that the grid was restored so quickly, and to my knowledge without any major injuries to the people responsible for putting it back together. Cudos to those responsible for a job well done. (also not on the front page)

I thought I would end on a positive note. Thanks for the opportunity to participate.

 

[SidiropoulosM]

etronics,

You say that &quot;I have enjoyed reading all the opinions and theories but unfortunately most of them miss the mark&quot;.

I have to disagree with you. This has been a superb thread ant the opinions expressed have covered a wide spectrum of possible causes and possible solutions.

...&quot;why there wasn't enough redundancy built into the system is also an issue that needs some investigation&quot;...

This is well known, that the investment in transmission infrastructure has not kept pace with the load growth. However, redundancy costs billions of dollars. The resulting increase in reliability may not be perceptible enough to consumers to justify their increased rates, which may double or triple. Furthermore, even with total redundancy you'd never be sure that another blackout will never happen. Human errors, acts of God, sabotage and other catastrophic events can occur and take out entire substations with cascading consequences.

&quot;Why other neighboring ISOs did not respond when they had working computer systems is another issue&quot;.

We don't know that other ISO's had sufficient information early enough to take timely action. They would if this were a frequency problem, as frequency would be the same everywhere in the system. But it seems that this was a voltage problem, which can be highly localized until the cascading takes place, when it's too late to do anything.

I agree with your last 2-3 paragraphs.

 

[lengould]

To those interested, I'd like to see some of you more technical types pick up again on the technical discussions, particularly re. Gunnar Englund - state machine monitor back on Aug. 18. I think it's not my place, Gunnar, but if you wanted you should start a new thread clearly marked &quot;experts only&quot; to persue further how a (relatively) small amount of money spent on [upgraded / improved / completely new] [concept controls systems and-or strategies / management decision tools / financial models / generation addition strategies] might enable the existing infrastructure to service requirements with fairly modest (e.g. as was in plans prior to blackout) additions to transmission capacity. I think you might consider starting several threads, e.g.

1) Realtime digital monitor/modeler/predictive analyser for large transmission grid.
2) Neural net / trainable software / expert systems predictive problem analyser (as operator assistant), how far ahead could it predict and with what reliability?
3) Standardized grid changes analysis (using historical data from 1?), analyses proposed load, gen, line and switch adds / deletes, recommends relaying / settings, communications delay times, islanding schemes etc.

I'd sure follow something like that closely, and promise not to post there.

Should thread &quot;Big Blackout. What Happened II&quot; also be started? Takes a long time to load this one now.

 

[RAMConsult]

I agree with lengould, it is time to start some new sub-threads. With all due respect to the previous posters, we need to maintain a focus on the original problem statement of &quot;...What Happened?&quot; As a power engineer, I'm less concerned with the 'coulda, woulda, shoulda' and more interested in gathering and analyzing the data that will help determine the actual sequence of events. I'm certain that during the process of discovery we will all get a finer appreciation of how electricity is generated, transmitted, distributed, and controlled. Armed with this fresh knowledge we will be in a much better position to influence the future.

 

[spitfire]

This thread now jumps to the Big Blqackout-What Happended II thread at:

thread238-70891