Looks like they lost the west 49th st substation. And reports of a manhole fire though Con-Ed says it may not be related. Any information on your side?
I agree with him and he is right when he says the system is specifically designed to prevent this. In fact the system can handle multiple failures (equipment failure + breaker failure) or failure of any two elements one after the other before load shedding or voltage reduction need to be implemented let alone a complete blackout.
There is also the fact relaying failed during Sandy at 13st st, and at Astoria 6 months ago. Several fires in the past few years at Farragut. Con Ed has some profound protective relaying issue if they are cropping up so often in such magnitude.
I'll predict there will be plenty of blame to go around. Antiquated infrastructure, lack of investment in redundancy, avoidance of disruptive construction projects, and of course, rats.
This is what I would bet it is. No one was providing oversight to the engineering because the utility was very short staffed or it was assumed the outside consultant would check their own work to make sure that it was always 100% correct. Problems like this accumulate and since faults are generally infrequent, it can take awhile for someone to notice that the relaying is not set correctly. Whatever the technical problem, in my opinion the causes is really a process and resource problem.
It takes a rare event to verify that your system is coordinated for a rare event. That isn't 100% true and some companies like Omicron have systems that take a system model and bombard the relay with a whole bunch of cases but I don't know of anyone that does that.
------------------------------------------------------------------------------------------
If you can't explain it to a six year old, you don't understand it yourself.
The authority rules out the following:
1) Overload
2) Ciber attack
3) Terrorism.
This site is a 40-year-old first generation of GIS with a large history of issues associated with leaking SF6 gas.
ConEd need to evaluate if this type of old installation with multiple feeders with oil-filled pipe-type cables should be adequate to maintain the reliability of the system expected for this application.
Below is an excerpt with public domain information for the 49th St Substation:
I hate to speculate, but could this be a system wide problem involving IEC-61850? This is really no different then the relaying failure in December. Twice primary and backup has failed in 6 months. Probability says this must be rampant.
Sounds like years of 'do more with less' cut maintenance and spend big on useless capital. These relay rooms probably flooded in the past and have never been properly recommissioned...
Don't think either of these stations have flooded that I am aware of. But yes, lack of maintenance could certainly be it.
Thinking this out it looks like not only did the relaying at 65th fail, but also West 49th in that 49th didn't just cut power to 65th but also 50th and 42nd which are not tied into the 5 138kv feeders serving 46th.
In theory a primary and secondary relaying failure should simply clear a 13.8kv bus section and one transformers (if a circuit switcher is present) at 46th on over current. Worse case scenario two transformers and one 138kv cable get tripped. The remaining load can be served via 4 cables and 8 transformers no problem.
But ok, all 5 138kv feeders trip. How do the reaming 6 coming out of 49th also trip?
There is a lot that went wrong here. If any of this involves microprocessor relaying the industry needs to know.
Concerning that there was such a massive failure of the protection system. All relays are digital? Complex relaying schemes are often not adequately tested for functionality - even if the protective element detects the fault, relay programming/configuration issues results in no breaker trip signal. Pointless speculation at this stage, I suppose.
I agree that it is somewhat pointless in that we do not have all the details on the relaying, but at the same time such a grand failure deserves a conversation. I know for a fact that if both primary and secondary 13.8kv relaying went at a 115kv-13.8kv substation around here it may (key word) clear the whole substation but not the entire 345-115kv bulk station supplying it.
However according to reliability reports, it appears that Time Square Network is fed via 50th st.
Anyone know if it was transferred over to another substation or has feeders from other networks? 50th also has the Hudson network, but it looks to have gone down. I'm beyond words confused.
Interestingly, or inversely, the Empire State building was supposed to be lit but was not. Looks like Con Ed mixed the networks/supply substation around a bit.
Here is Murray Hill supplying Fashion and Empire at the time of Hurricane Sandy.
Networks supplied by Murray Hill in 2015. The number of transformers mentioned in the table are those with PQ monitoring, not the total number in the substation or network.
Murray Hill fed by Vernon at the time of Sandy.
The blue line represents 5 138kv feeders originating from Vernon supplying East 40th st and Murray Hill substation.
If wondering Queens bridge takes 138kv and steps it down to 69kv with 7 69kv feeders supplying East 63rd st.
The Vernon double ring bus, which interconnects generation to the 138kv transmission system in Queens.
38M01, 38M03, 38M04, 38M05, 38M06 feed out to 40th st and Murray Hill.
My guess for the numbering (missing 38M02) is that at one point 7 69kv cables fed out but were upgraded to 5 138kv cables.
If I have it right 38M72 is no longer "capped off" (alive on back-feed) but connected through a phase angle regulator were some power can be exchanged between West 49th 138kv and Vernon.
“Out of an abundance of caution, we have taken preventive measures by isolating similar relay equipment at other substations,” Con Edison added. “We will analyze and test the equipment before we put it back in service. Our electrical delivery system continues to operate with multiple layers of relay protection.”
