Controllogix 1756-L63 Redundancy Type 12 Error

[AndyHill]

We are using a pair of 1756-L63 Proccessors set up with 1756-SRM redundancy module.

main racks constists(2): CPU, Controlnet, ENet, SRM
I/O Racks constists(2) : Controlnet,ProsoftModbus Card,SPare,Din,Din,Dout,Ain.

One of our hardware techs found out that if the Slave CPU is switched of when "DISQ" and then swithced back on first(as the only CPU on the netwrok). We genetrate a major fault of type 12 code 32 ( Secondary became master) and processor haults.

So he switched off the slave and switched on the other proccessor. It booted back up and assumed because it had orginally been master and was the now the only CPU on the network, the roll of Master. He then switched on the faulted slave and it stays faulted and DISQ.

The only way to clear this faulted slave is log in with RSlogixs 5000 to the secondary IP (master+1) and press the clear amjor faults button.

What we want to do is
1) Stop the CPU's from Generating this error and getting into this state. With the Orginal Master remaining Master. (we have changed procedures so this should happen but a procedure is not fail proof).
OR
2) If the secondary CPU starts faulted get the running Primary to issue a CIP message to the secondary CPU to clear Fault table.


So far :- (none work)
1)I have been speaking to A/B support about this but no result yet.
2) I have "Wireshark" sniffed the CIP messages generated when you press the button in RS Logix and successfully got the Master CPU to generated the same message but the Slave doesn;t respond correctly (i.e. clear its fault).
3) Tried various fault handlers to clear Fault on power up ( Controller Handler, Power Handler & Program Handler)



Firmware : 1756-L63 CPU 16.54
SRM 5.2.3
Linx Classic 2.52
RSLogix 5000 ver 16.03

 

[jde777]

I would use rslinx to browse each device and get the properties and make sure all firmwares in the pri and sec chasis are in fact identical.

 

[AndyHill]

That has already been done.

 

[controlsdude]

Looks like your on CPR7 release on firmware/software. Looks ok.

What I have done in the past with a logix generated fault is to put piece of code in the fault handler to handle this when you get a know good error. There is probably a bad redundancy error. This would keep you in run mode and still know it went to the backup processor.

I know that when you share data between processors to get the speed on changover, all dints have to be in same file, all ints in same file, all bools in same file. Is this the issue just too slow in changover? Was it a watchdog timeout on the changover?

 

[AndyHill]

Speed of change over is fine, the problem occurs when we shut down our sysem as the network switches go down first then the CPU's this kills the redundancy then when you bring the system back up you may or may not get this error depending on which cpu starts first.
Usually this error occurs on CPu in Chassis 1 is it was DISQ on power off this is because A/B weighting tells the lowest chassis to assume master when both are powered on. However I feel this is in correct because the lowest chassis was DISQ on power off.

To generate error follow these steps
1) Sync System.
2) Disconnect Network Cable on Primary (A)
3) Switch off (A)
4) Switch off (b) This will be PRIM
5) Reconnect Cable to (A)
6) Switch on (A)
7) (A) stops with a Type12 Code 32

So start recovery as we switched on the DISQ PRIM be mistake
8) Switch off (A) (if we didn't switch off (A) then B would become secondary as DISQ
9) Switch on (b) This remains PRIM
10) Switch on (A) - A remains FAULT and DISQ.

 

[jde777]

What does the CNBR display when this occurs?

 

[jde777]

Also have all modules in the system been checked for the Atmel issue?

http://rockwellautomation.custhelp....ion.cfg/php/enduser/std_adp.php?p_faqid=41204

 

[djs]

Chapter 9 of the RsLogix 5000 System Reference manual describes a procedure to reset a major fault under program control. Perhaps this will help.

 

[AndyHill]

DJS- I have been through the Chapter 9 of the 5000 Ref Manual and this does not cover type 12 and doesn;t tell you which handler to use for type 12. Also speaking to Rockwell UK ( they as speaking to rockwell USA).

 

[EddieWillers]

I was able to re-create the failure following your steps.

Once you re-power both controllers and clear the fault with they keyswitch, the redundant pair takes off and runs again.

I suspect the powerup handler is going to be the way around this issue. I'm interested in what RA Tech Support has to say about it.

 

[AndyHill]

Eddie, How did you manage to clear the fault with the Key switch?
Did you check using RS logix was it the type 12 code 32 ?

So far rockwell are saying - Which won;t work as we need to be able to save the parameters.......


QUOTE -

"' When a disqualified processor is powered down... it knows it was not the primary
and not qualified, and has no guarantee that it has the current program that was in
the primary... what happens if someone installed some critical edits into the primary
since the rack disqualified... or even downloaded a totally different program to the
primary... it may no longer have the intended current program and/or values. This
fault is a safety feature to prevent the controller from running old code or even a
different project then intended. There is no way to programmatically clear this fault,
manual intervention is required. The intention is that the customer could then
determine if the application in the controller is still the one intended.
The only option I can see to prevent this is the use of the CF64 card. They can
burn the program to the card, then tell it to load on every power up. This will bypass
the fault. The only drawback is if they change the program, they will need to
reprogram the CF card in each processor after the changes. Data values since the last
store to the card will be lost. If they have changing values that must be maintained,
they would need an HMI or some device to write the last read values.
If they do not change the program, and do not need to save tag values between
powercycles, and do not mind reverting to a snapshot in time of the program, then the
CF64 card would probably be the easiest option. Is this a possibility for the customer
?
"