Eng-Tips is the largest engineering community on the Internet
Intelligent Work Forums for Engineering Professionals
Manhattan Partial Blackout 2
- Thread starter Mbrooke
- Start date
thermionic1
Electrical
Back in the day, the PRSC used to maintain a list of know relay issues. I don't think it gets updated anymore. The mention of the interface between the "sensor" and the relay sounds like a terminal block or maybe a test switch?
I do recall seeing a demo of the GE "Brick" (61850 type merging unit), maybe it was 2008?
Utilities in my region tend to take a "wait and see" approach to newer technologies. 61850 is in that list as well as the new SEL traveling wave relay.
If this is some kind of 61850 related failure it would give all the ammunition in the world to the skeptics and ultra conservative practices seen in many utilities.
I do recall seeing a demo of the GE "Brick" (61850 type merging unit), maybe it was 2008?
Utilities in my region tend to take a "wait and see" approach to newer technologies. 61850 is in that list as well as the new SEL traveling wave relay.
If this is some kind of 61850 related failure it would give all the ammunition in the world to the skeptics and ultra conservative practices seen in many utilities.
- Thread starter
- #22
It certainly would, and the manufacturer involved.
From another forum I learned that Con Ed disclosed what happened but sadly nothing released to the public:
I think journalists really need to get involved because if people knew what I suspected there would be a lot of outrage. It is my belief that are hundreds if not a thousand relays with several latent software defects out on the Con Ed system.
From another forum I learned that Con Ed disclosed what happened but sadly nothing released to the public:
I think journalists really need to get involved because if people knew what I suspected there would be a lot of outrage. It is my belief that are hundreds if not a thousand relays with several latent software defects out on the Con Ed system.
A colleague commented:
A good design should include primary and backup protections devices from different technologies. It is the case, why did this flaw happen?
Another factor might be associated with the O&M program and the qualification and training of the personnel. It is well knowing the downsizing and retirement of the labor forces and qualified knowledge engineers. ConEd is notorious for hire temp. and cut in training.
In my opinion, the root cause official note is vague and lock of credibility. For the high rate of energy in NYC, the customer and the general public deserve a better and honest report indicating the real root cause of the frequent power outages.
A good design should include primary and backup protections devices from different technologies. It is the case, why did this flaw happen?
Another factor might be associated with the O&M program and the qualification and training of the personnel. It is well knowing the downsizing and retirement of the labor forces and qualified knowledge engineers. ConEd is notorious for hire temp. and cut in training.
In my opinion, the root cause official note is vague and lock of credibility. For the high rate of energy in NYC, the customer and the general public deserve a better and honest report indicating the real root cause of the frequent power outages.
- Thread starter
- #24
What do they mean by "flowed" connection?
IMO I still think they are hiding a communications failure triggered by bad software in a major relay upgrade.
Just to give you an idea Con Ed has really been hitting IEC-61850 and fiber optics:
Something I do not agree with. And is just a feel good for flood prone substations.
Other countries are building plenty of brand new substations and for good reason everything is hard copper. No bricks, merging units, optical CTs, ect for the tripping and local monitoring.
IMO I still think they are hiding a communications failure triggered by bad software in a major relay upgrade.
Just to give you an idea Con Ed has really been hitting IEC-61850 and fiber optics:
Something I do not agree with. And is just a feel good for flood prone substations.
Other countries are building plenty of brand new substations and for good reason everything is hard copper. No bricks, merging units, optical CTs, ect for the tripping and local monitoring.
HamburgerHelper
Electrical
There turns every day with networked solutions and you don't have to try hard to find an example of something going wrong due to miswiring.
------------------------------------------------------------------------------------------
If you can't explain it to a six year old, you don't understand it yourself.
------------------------------------------------------------------------------------------
If you can't explain it to a six year old, you don't understand it yourself.
- Thread starter
- #26
But you can test wiring- thats the thing.
SEL has the right idea at least:
SEL has the right idea at least:
- Thread starter
- #27
This is getting very interesting.
This I team investigation makes a very good case that old outdated electromechanical relays may have been the culprit:
-However-
Here they talk about "Software" leading me to think they are talking about a new microprocessor device:
Me thinks its time folks file some freedom of information forms.
I know public documents have been far more transparent in the past up to including the make and settings of the equipment ie, for example the Astoria generation project relaying.
This I team investigation makes a very good case that old outdated electromechanical relays may have been the culprit:
-However-
Here they talk about "Software" leading me to think they are talking about a new microprocessor device:
Me thinks its time folks file some freedom of information forms.
I know public documents have been far more transparent in the past up to including the make and settings of the equipment ie, for example the Astoria generation project relaying.
My experience has been that most relay manufacturers are not very forthcoming in advising customers of known issues with their devices. SEL is generally pretty good. ABB has been NOT good, at least in my limited experience. I'd put GE/Multilin somewhere in-between.
- Thread starter
- #29
As an example of the typical redundancy in a Con Edison area substation and network, here is how a typical 13.8kv supply substation is configured which should also hold true for 65th st.
I am near 100% certain the substation in figure 1 is East 63rd st (half of it anyway).
I'd be surprised if there wasn't directional over current on the 13.8kv transformer breaker and synchronous bus breaker as a last resort backup to failed feeder relaying. I mean there really should be. The loss of a transformer and bus section at 63rd would not result in any lost load as all Manhattan networks can handle the loss of two feeders at peak load. It has to be the same for the 65th station. Even if a lost bus section at 65th st removes more then two feeders a network should still be able to survive that considering it was not a high peak demand period before the blackout.
The networks fed by East 63rd st (ignore the first two columns on the left, and make note that they do not refer to the number of transformers at the substation or in the network but rather the number of transformers with PQ monitors)
Hunter, Sutton, Turtle Bay and Roosevelt each have 12 feeders.
7 69kv feeders from Queens Bridge come into East 63rd st. Each 69kv feeder supplies two transformers or 14 in total. 7 supply 13.8kv feeder switchgear in part 1 of East 63rd st while the other 7 supply 13.8kv feeder switchgear in part two of East 63rd st.
One "part" is shown here:
Each 13.8kv main bus section has at least 4 feeder breakers, two feed one network and two feed another network.
A bus fault or feeder breaker failure results only in the clearing of 4 feeders, or two in each network.
The loss of a 69kv transmission supply feeder results in two transformers de-energizing- or one in each part of the substation.
This again is not problem as the transformer breaker senses the reverse power and opens.
In fact you could loose two 69kv feeds- 4 transformers (two in each part of the station) and not have outages.
46 MVA x 5 = 230 MVA, or 460 MVA for the whole station.
All in all the design is very redundant and over built.
Reason I am using 63rd as an example is because its not far off from most other substations in Manhattan like such as those involved in the blackout.
To give you an idea, here are the transformers at 63rd st. The colored "tubes" contain the 69kv feeds to the trafo primary.
14 arrows in total.
Another shot, the tree growing in the middle covers one of them up.
The other grates cover capacitor banks and similar equipment.
West 65th st involved in the blackout.
10 transformers, two 65MVA transformers per 138kv cable, 5 per section run in parallel.
Same operating principal.
I am near 100% certain the substation in figure 1 is East 63rd st (half of it anyway).
I'd be surprised if there wasn't directional over current on the 13.8kv transformer breaker and synchronous bus breaker as a last resort backup to failed feeder relaying. I mean there really should be. The loss of a transformer and bus section at 63rd would not result in any lost load as all Manhattan networks can handle the loss of two feeders at peak load. It has to be the same for the 65th station. Even if a lost bus section at 65th st removes more then two feeders a network should still be able to survive that considering it was not a high peak demand period before the blackout.
The networks fed by East 63rd st (ignore the first two columns on the left, and make note that they do not refer to the number of transformers at the substation or in the network but rather the number of transformers with PQ monitors)
Hunter, Sutton, Turtle Bay and Roosevelt each have 12 feeders.
7 69kv feeders from Queens Bridge come into East 63rd st. Each 69kv feeder supplies two transformers or 14 in total. 7 supply 13.8kv feeder switchgear in part 1 of East 63rd st while the other 7 supply 13.8kv feeder switchgear in part two of East 63rd st.
One "part" is shown here:
Each 13.8kv main bus section has at least 4 feeder breakers, two feed one network and two feed another network.
A bus fault or feeder breaker failure results only in the clearing of 4 feeders, or two in each network.
The loss of a 69kv transmission supply feeder results in two transformers de-energizing- or one in each part of the substation.
This again is not problem as the transformer breaker senses the reverse power and opens.
In fact you could loose two 69kv feeds- 4 transformers (two in each part of the station) and not have outages.
46 MVA x 5 = 230 MVA, or 460 MVA for the whole station.
All in all the design is very redundant and over built.
Reason I am using 63rd as an example is because its not far off from most other substations in Manhattan like such as those involved in the blackout.
To give you an idea, here are the transformers at 63rd st. The colored "tubes" contain the 69kv feeds to the trafo primary.
14 arrows in total.
Another shot, the tree growing in the middle covers one of them up.
The other grates cover capacitor banks and similar equipment.
West 65th st involved in the blackout.
10 transformers, two 65MVA transformers per 138kv cable, 5 per section run in parallel.
Same operating principal.
-
1
- Thread starter
- #30
While piecing together the incident, I can confirm that from a hardware perspective the system is indeed designed as a second level contingency system.
Each area station has 5 or 10 65MVA transformers with no more then two sharing a 138kv supply feed.
65th St:
Astor:
West 42nd:
West 49th and West 50th st. Yellow arrows show 420MVA 345kv to 138kv auto transformers also based on a second level factor. Red arrows show 65MVA 138kv to 13.8kv units.
Stations supplied by West 49th autos:
345kv-138kv auto transformer ratings:
138kv-13.8kv transformer ratings:
If any 138-13.8kv transformer needs to be de-energized for an uncleared secondary fault only one 138kv cable is tripped. In the case of Astor it will result in a circuit switcher trip, as for the other 3 stations I am not sure but I'd say a cable trip due to their age (hence no circuit switcher).
A 138kv cable clearing will simply result in one auto transformer and 6 out of 30 138kv-13.8kv transformers being cleared. The remaining 4 autos, 138kv cables and 24 area substation transformers would remain energized and due to their 13.8kv secondaries being in parallel absolutely no network load would be lost.
The assumption above is made that the auto transformers feed the 138kv cables directly with at most a disconnect between them.
Even if the 5 auto transformers feed an elaborate 138kv ring bus at 49th st- the fault clearing topology would still remain the same and thus the outcome. Only the breakers around (for) the 138kv cable requiring de-energizing would open.
I can only conclude that:
1) In addition to the relaying failures at West 65th relay misoperations also took place at 49th st.
2) West 65 lacks directional over current or any over current for that matter on its 13.8kv transformer breakers. The uncleared 13.8kv feeder fault presented itself as an overload condition. Eventually some device caught the 5 paralleled transformers over heating and issued a trip command for all 5 clearing all the 138kv cables and the area substations they supply. But I can't see not having any 50/51 protection with secondary trip on the 65MVA units.
I'm left with more questions then answers.![[3eyes]](/data/assets/smilies/3eyes.gif "[3eyes] [3eyes]")
![[ponder]](/data/assets/smilies/ponder.gif "[ponder] [ponder]")
Each area station has 5 or 10 65MVA transformers with no more then two sharing a 138kv supply feed.
65th St:
Astor:
West 42nd:
West 49th and West 50th st. Yellow arrows show 420MVA 345kv to 138kv auto transformers also based on a second level factor. Red arrows show 65MVA 138kv to 13.8kv units.
Stations supplied by West 49th autos:
345kv-138kv auto transformer ratings:
138kv-13.8kv transformer ratings:
If any 138-13.8kv transformer needs to be de-energized for an uncleared secondary fault only one 138kv cable is tripped. In the case of Astor it will result in a circuit switcher trip, as for the other 3 stations I am not sure but I'd say a cable trip due to their age (hence no circuit switcher).
A 138kv cable clearing will simply result in one auto transformer and 6 out of 30 138kv-13.8kv transformers being cleared. The remaining 4 autos, 138kv cables and 24 area substation transformers would remain energized and due to their 13.8kv secondaries being in parallel absolutely no network load would be lost.
The assumption above is made that the auto transformers feed the 138kv cables directly with at most a disconnect between them.
Even if the 5 auto transformers feed an elaborate 138kv ring bus at 49th st- the fault clearing topology would still remain the same and thus the outcome. Only the breakers around (for) the 138kv cable requiring de-energizing would open.
I can only conclude that:
1) In addition to the relaying failures at West 65th relay misoperations also took place at 49th st.
2) West 65 lacks directional over current or any over current for that matter on its 13.8kv transformer breakers. The uncleared 13.8kv feeder fault presented itself as an overload condition. Eventually some device caught the 5 paralleled transformers over heating and issued a trip command for all 5 clearing all the 138kv cables and the area substations they supply. But I can't see not having any 50/51 protection with secondary trip on the 65MVA units.
I'm left with more questions then answers.
- Thread starter
- #31
- Thread starter
- #33
Two papers that may shed more light on the event- or at least a bit on the protection system/philosophy itself.
Interestingly they do not note the 138kv-13.8kv transformers as having over current protection.
And to be clear regrading the "wave" in the 138kv cable I highly doubt thats actually a fuse.
Page 7:
Paper #2:
Interestingly they do not note the 138kv-13.8kv transformers as having over current protection.
And to be clear regrading the "wave" in the 138kv cable I highly doubt thats actually a fuse.
Page 7:
Paper #2:
- Thread starter
- #34
Lastly I just want to add an approximate single line diagram of the West 65th substation. It came out to big so I had to resize it and cut it into two- my apologies for resulting blurriness.
- Thread starter
- #35
September 3rd the state legislature will hold a hearing on both blackouts:
I hope on Con Ed can explicit even if it means revealing latent defects within the specific device(s) and manufacturer involved.
I hope on Con Ed can explicit even if it means revealing latent defects within the specific device(s) and manufacturer involved.
-
1
- Thread starter
- #36
Watching the vid presently... Now they are saying that transformer relaying maloperated at 65th st but somehow the 13kv did trip...? So if a trafo protection relay mal operated why didn't the transformer circuit switcher just trip? And how did one transformer relay maloperation result in all 5 transmission supplies tripping? And 3 systems malfunctioned? Last one I can believe.
There is absolutely no information being disclosed that answers any of the original questions. So much was said without actually saying anything.
If you want my opinion it wasn't one or two relays that malfunctioned, but at half dozen or more, at two substations. There is a lot they are hiding. A lot.
- Thread starter
- #37
Before I forget I just want add (stress) why this is so big- a substantial concern for an area substation triggering multiple transmission shutdowns is not just in local outages but potential system stability/thermal issues. Which is why NY-ISO is also investigating the West Side incident.
A majority of 345-138kv and 138-69kv auto transformers do not have secondary breakers and rely on their primary breakers to de-energize low side cable faults and DTT events. Because transmission cables and auto transformers alternate every other position on Con Edison's ring buses, typically between half and all transmission cables are lost when an area substation calls for the tripping of all its supply lines.
Similarly major 138kv stations alternate between lines, generators and load serving transformers. Tripping of all 5 transformers or all 5 radial area substation feeds results in at least half of all lines and gens being cleared.
Typically this puts the system beyond its normal design contingency necessitating immediate action.
Ie, clearing of all cables in a major 345kv path:
Clearing of two 345kv cables:
Wanted to make a single line of Farragut to better illustrate it but did not have time. A total area substation trip removes 5 345kv circuits on one side of the station.
Rainy:
Queens Bridge:
Flow paths that can be effected.
A majority of 345-138kv and 138-69kv auto transformers do not have secondary breakers and rely on their primary breakers to de-energize low side cable faults and DTT events. Because transmission cables and auto transformers alternate every other position on Con Edison's ring buses, typically between half and all transmission cables are lost when an area substation calls for the tripping of all its supply lines.
Similarly major 138kv stations alternate between lines, generators and load serving transformers. Tripping of all 5 transformers or all 5 radial area substation feeds results in at least half of all lines and gens being cleared.
Typically this puts the system beyond its normal design contingency necessitating immediate action.
Ie, clearing of all cables in a major 345kv path:
Clearing of two 345kv cables:
Wanted to make a single line of Farragut to better illustrate it but did not have time. A total area substation trip removes 5 345kv circuits on one side of the station.
Rainy:
Queens Bridge:
Flow paths that can be effected.
Palletjack
Electrical
I looked and I see disconnects, the breakers are on the ground. I saw on your one lines from the August 2 post a maintenance scheme. The 6 transformers on the edge feed the load until the one in the middle needs to carry load to perform maintenance on one of the perimeter transformers.
- Thread starter
- #39
From what I've figured there are no breakers on the ground- the disconnects feed potheads transitioning to underground cables.
East 13th st is one of the few that has breakers on its outgoing area substation feeders, with Astoria West and Vernon for its generators.
A few more pics of Farragut and Queens Bridge showing the cables heading into the earth:
East 13th st is not a concern for a total transmission outage making an impact beyond directly resulting load loss:
East 13th st is one of the few that has breakers on its outgoing area substation feeders, with Astoria West and Vernon for its generators.
A few more pics of Farragut and Queens Bridge showing the cables heading into the earth:
East 13th st is not a concern for a total transmission outage making an impact beyond directly resulting load loss:
- Thread starter
- #40
Con Edison had qualms about loosing an entire area substation. At first I was unsure whether or not 65th st had circuit switchers, but after reading through commission filings they should have been in place at the time of the incident.
Looking at Google maps taken in 2017 "substations beyond 2018" already had/have circuit switchers installed:
The event which started the program:
The result of Seaport clearing all 5 138kv feeders also de-energized the Trade Center substation and I would imagine took out 5 345kv feeders at Farragut which would certainly alarm Con Ed / NY-ISO on a hot summers day.
At the time Seaport fed the Fulton network, but Trade Center going down also removed the Park Place network, Battery Park network, World Trade Center network and the Cortlandt network. So outage was substantial.
Looking at Google maps taken in 2017 "substations beyond 2018" already had/have circuit switchers installed:
The event which started the program:
The result of Seaport clearing all 5 138kv feeders also de-energized the Trade Center substation and I would imagine took out 5 345kv feeders at Farragut which would certainly alarm Con Ed / NY-ISO on a hot summers day.
At the time Seaport fed the Fulton network, but Trade Center going down also removed the Park Place network, Battery Park network, World Trade Center network and the Cortlandt network. So outage was substantial.
- Status
Similar threads
- Locked
- Question
- Locked
- Question
- Locked
- Question