Digital Relay Loosing it’s Mind 3

[DTR2011]

We’re currently commissioning a large substation with A and B systems from different manufacturers. The P&C design has done away with control switches, Lock out relays and replaced with virtual lockouts.

We’ve discovered that on the B system, the non volatile latches, used as 86 devices, are reprogramming themselves to be not used. Contact with the manufacturer has revealed that the problem is repeatable.

This is the first time I’ve experienced anything quite like this. The manufacturer has not offered anything else to try other than to up/down grade firmware, but that hasn’t solved anything and requires a full retest. Due to the complex logic involved, replacing control switches and LORs it takes hours to retest.

The B system was deemed Good Enough to use in the design but that may change shortly.

 

[davidbeach]

Real, physical, switches exist for a reason. Just because it's possible to virtualize them doesn't make that a good thing to do.

It would be useful to know that the "B" system is. Without knowing that it's not possible to provide much other assistance.

 

[DTR2011]

I fully agree with switches, LOR’s, etc. Add in 61850 and all kinds of apparatus monitoring and alarms that WILL get screwed with some how.

All of the logic that could have been proven with a fluke now requires a logic test with SER for each go & no go.
Not my design.

UR’s with ver 7.9x and 8.03 FW.

 

[davidbeach]

It's impossible to tell what goes on inside a black box... ;-)

 

[DTR2011]

We are finding that out every step we take in trying to resolve this issue.

It was pure coincidence that this was found in the first place. Imagine setting up a relay, testing it for 8-12 hours and maybe tomorrow SOME of the latches fail to operate but the corresponding LED says it is....

 

[RRaghunath]

UR family belongs to GE Multilin, one of the best in protection relays in the market.
I am surprised they are not able to solve the issue!!

 

[DTR2011]

We are actively working with the manufacturer to resolve the issue. They have been able to replicate the issue with our settings at the factory. There was a fairly serious meeting with top management of the utility and OEM. So far none of the suggestions from the manufacturer have worked.

A large station has to be energized in about a month and whatever resolution is made, about 15 devices will have to be completely retested after already having been fully tested. It’s 6-8 hours per device with all of the internal logic being used. We’re a service provider, so we’ll be compensated, that’s not the concern. Our schedule has already been severely delayed due to Covid, with the in service date not changing.

 

[LionelHutz]

Multilin was one of the best protection relays. Adding the GE made that questionable. GE thinks they are made of gold though.

It's good to know about this because I'd be more likely to use their new 8 series relays with similar internal logic and they likely borrow or share code with the UR relays.

 

[DTR2011]

Well we think we found the culprit...

The site uses SEL Team SW in the station computer. Team uses Modbus to retrieve the event reports for the UR relays. Apparently, GE changed the Modbus addressing at some point, so that what should be a data retrieval register is now a write register, hence the NV latches were being written over by file retrieve instructions.

Disabling the Team polling of the UR devices is a short term fix. I’m not quite sure why it’s necessary to poll both A&B systems given there is engineering access to begin with.

There appears to be other ways (FTP, Y modem) of getting event reports.

I would not want to show up to a blackout station and try and troubleshoot this at in two in the morning. I feel sorry for the local techs that have inherited this Rube Goldberg system.

 

[stevenal]

NERC lesson learned? This should be shared more widely than just Eng-Tips.

 

[FreddyNurk]

I would have thought that for file retrieval the Modbus instructions are effectively read only (i.e. if I recall correctly read and write are two separate modbus functions, and its entirely possible to read writable registers as well as write to them), and as such enabling write access on specific registers would have no effect if no write functions were being used for the retrieval.

That to me sounds like the internal relay modbus configuration is borked. But yes, if there's better means (engineering access, SFTP / FTPS) to get reports that would also be a better move.
I presume that the relays and / or team software don't support DNP3?

EDMS Australia

 

[LionelHutz]

I would agree with that. I would really hope the GE relays don't require a Modbus write to read data from them. That's a stupid implementation just asking to be broken. I would also hope Teams doesn't write when it's not necessary either.