Emergency Stop button not working 10

[Skogsgurra]

There's this saw mill where the safety is built around gates, light barriers and quite a few red mushroom style emergency buttons that are connected via several AS-interface buses, around ten of them.

We were asked to go and have a look at the mill. There were a multitude of reasons for that. One of the main reasons was that there were a lot of nuisance stops. We traced the stops to the AS-i buses and the interference level present on them (PWM drives in hundreds) with motor cables running parallel to the yellow bus cable. The yellow cable was in separate and well grounded trays, but the interference was still quite high. After connecting the GND terminal of the Masters to ground (an oversight from the panel builder and commissioner), the interference level was a lot lower and the bus worked a lot better.

Then, we noticed that there were Emergency Stops that didn't work at all. They had been crossed out in the PLC program, but left for anyone to see and press in the plant. Someone had given up and just left the saw mill with the Emergency Stops in non-working order. There wasn't even a sign or a note about the non-functioning devices.

After this long introduction, the question is short and simple: On a scale running from Stupid via Careless and Irresponsible to Criminal - where would you put such a behaviour?

The second question is also simple: What about OSHA and legislation? What about laws in EU? I have never seen anything like this before and welcome any view on the matter, be it emotional, factual or whatever.

Gunnar Englund

http://www.gke.org

--------------------------------------
Half full - Half empty? I don't mind. It's what in it that counts.

 

[flexoprinting]

I don't understand how you can override some stop buttons and not all in a PLC, normally they are wired in series to one input, but since they are crossed out in program, the manufacturer designed them to work and most likely some careless Tech under pressure for production jury rigged the machine. Unethical, criminal, and careless for sure.

In the US by OSHA rules I believe, Employees are required to report safety issues and can refuse to work under unsafe conditions, report the condition to OSHA for immediate investigation and correction.

 

[Skogsgurra]

No, they are not at all wired in series. Have a look at Jokab Pluto. Every button has a unique address in the PLC program and this is (SW)SIL stuff. So, the behaviour is really very remarkable. The buyer wanted maximum safety and he got none at all. At least in some parts of the mill.



Gunnar Englund

http://www.gke.org

--------------------------------------
Half full - Half empty? I don't mind. It's what in it that counts.

 

[jraef]

It's stupid, until someone is injured or killed, at which time it becomes criminal.

My very first engineering job was investigating a project that went horribly bad and resulted in lawsuits. The project was for an OEM of cardboard compactors, my new employer had (prior to my joining) sold a control panel to them, but with no wiring, they just mounted the components because the OEM wanted to save money by wiring them himself. He somehow managed to wire the power leads feeding the auger hydraulic pump from the wrong side (line side) of the main disconnect. So true to Murphy's Law, a grocery store employee was tossing boxes into the hopper to be compacted, saw something he wanted, reached in to rescue it and got his sleeve caught in the auger. It was moving slowly so his screams for help got someone there on time, who pulled the disconnect to kill power. Unfortunately because of the wiring error, the hydraulic pump was NOT stopped and it tore the guys arm off. Stupid cannot do justice to the tragedy as it related to the guy who willfully disregarded simple safety protocols, the compactor OEM eventually was jailed for reckless endangerment. Stupid turned out to be my employer selling a "control panel" with their name and UL label on it without having done the actual wiring design or fabrication. The lawyers went after the deepest pockets.

"Will work for (the memory of) salami"

 

[ScottyUK]

Somewhere at the reckless and dangerous end of the spectrum. The designer and the commissioning guys should have some awkward questions to answer, but not half as awkward as those that the facility operator faces if someone is hurt or killed.

I wouldn't be happy with a comms-based ESD system, SIL or otherwise. Call me old fashioned.

 

[controlsdude]

Asi Estop, my take on Comms based estop networks is that basically for them to work they have to be separate network from the controls. The people I talked with said you can do it with the controls network but you have to do a bunch of checks to ensure the sil level. I came to conclusion that creating an estop comms network was the only way to ensure it worked correctly. This went for the AB type Ethernet or devicenet estop type networks. I know profinet was sold this way too but again don't believe the reps when they say you can have both control devices and estop sil devices on same network.

 

[ScottyUK]

skogs,

What was the response of the controller to loss of comms from an E-Stop? I would expect that if a device was polled by the controller and it failed to respond within the expected time then the controller should fail safe, i.e. initiate a trip. Was this the source of the nuisance tripping?

Does the network architecture allow you to use concentrators or hubs to gather a number of E-Stops in physical proximity to each other and then connect the hubs back to the controller by optical fibre? That might give you a chance of controlling the interference by breaking the network into smaller galvanically isolated elements. I've done this using other network types when I have had very bad interference problems but I know next to nothing about AS-i bus. I'm dimly aware that it might be a Siemens product.

 

[itsmoked]

I suspect Scotty is right and so the 'culprit' had to disable those various E-stops to get the plant mostly moving. Still mega-stupid.

Keith Cress
kcress -

http://www.flaminsystems.com

 

[Skogsgurra]

I simplified the situation a bit. The colleague that does this part of the investigation would probably express himself more correctly. The Jokab/ABB system is a very good one and it is built with an architecture very similar to what Scotty describes when saying "concentrators or hubs to gather a number of E-Stops in physical proximity to each other and then connect the hubs back to the controller by optical fibre". The problem here is that someone didn't realise that a non-functioning Emergency Stop is a violation of the safety thinking that is prevalent in the mill and that is expressed in the Machinery Directive 2006/42/EC and the Control systems safety standards EN ISO 13849-1 and EN 62061.

One could see it as an excusable "slight oversight", but I cannot do that; one slight oversight, two oversights, gross ignorance... The problem is known from nuclear power plants and oil fields - with known consequences. There is no place to draw the line other than below zero oversight.


Gunnar Englund

http://www.gke.org

--------------------------------------
Half full - Half empty? I don't mind. It's what in it that counts.

 

[Marke]

I would classify this a b... dangerous and pull the fuses until it was fixed!!
Commonly, programmers today have never been in the real world and seen what happens when things go wrong.
Recently had an argument with a bunch of programmers who maintained that fail-safe meant that the motors kept running if the comms was lost. Reason, if the motor happened to be in a tunnel and there was a fire that burnt through the comms cable, then it may be important that the fans kept running, so that was fail safe!!
Would not accept that there were fire mode specs that were separate to fail safe operation expectations.
Gunnar, I would get someone to check that the motor cables were a) screened and b) the screens are correctly clamp terminated at each end. This usually sorts these types of problems.
Best regards,
Mark.

Mark Empson
Advanced Motor Control Ltd

 

[LionelHutz]

Beyond the fact that what you describe runs somewhere between careless and criminal, I would also think the plant has a poor safety culture if they were operating without the E-stop buttons operational. Did this plant spend a whole bunch of money to implement a fancy new safety system but not be bothered to spend the time to create the proper safety procedures and ensuring their employees understand that safety is a top priority? Part of the safety procedures or safety culture should have been testing the E-Stop and other safety systems on a regular basis. The best safety systems are no good if they're not embraced by the employees.

 

[waross]

I feel your pain Gunnar. If someone is injured or killed, then someone may go to jail, but probably not until the damage is done.
I am presently on a large project which will soon be going online. There are about 1000 electricians on site. The LOTO (Lock Out/ Tag Out) procedures are rigorous. Everything is documented. There is a paper trail to prove conclusively that a circuit is safely locked out before work starts.
Unfortunately there is no procedure in place to verify that the CORRECT circuit has been locked out.
I have been suggesting that point to point testing be mandatory as part of the LOTO plan.
Management doesn't feel that point to point testing is required.
I have seen several LOTO violations.Some have been serious violations.
Just one example. There have been others.
A crew was about to work on a panel and went through the LOTO procedure correctly. They then went to the panel and fortunately did a test before touch. The panel was hot! 600 Volts. The panel had been energized by a temporary feed.
Discussion at a safety meeting.
Me;
"We were very lucky. Suppose that the temporary feed had been switched off when the crew did the test before touch test. They would have started work and the current may have been turned on at any time."
The voice of hubris;
"Well any time that you don't feel safe, you have the right to decline the work." (Ya right, that would be a good career move in this culture!)
"I am not worried about myself. I am worried about the younger, less experienced workers. And Sir, I am worried about your liability position should someone be injured due to a flawed LOTO system .
Please consider, had that circuit been switched off and the men started work, and then the power came on and someone was killed. You may be in more liability than you care to think about."
End of meeting.
If I was higher up the food chain that would have been my last day on that site!
The head of our companies safety division on this site supported me. He has been transferred to another site.
I'll probably be laid off eventually if I don't shut up.
It won't be the first time.
The last time it happened, I was exonerated by the grievance procedure and at least one high placed supervisor lost his job.
I was luckier than Cass that time.


Bill
--------------------
"Why not the best?"
Jimmy Carter

 

[racookpe1978]

It's counterintuitive, but you CANNOT verify a 'typical, real-world" power panel by tracing cables.

You LOTO process is NOT safer by requiring that cables and wires be physically traced. The ONLY way to verify the panel is de-energized is to open the panel wearing the proper PPE using the proper. Then, once within the panel, you MUST use prudent safeguards and procedures to verify by probe and voltmeter that the leads themselves are dead.

It is physically IMPOSSIBLE to trace cables from a panel back to the trays and back to their assumed power source.

 

[crshears]

Hey Bill,

You wrote: "I have been suggesting that point to point testing be mandatory as part of the LOTO plan. Management doesn't feel that point to point testing is required."

The term 'point to point testing' I have always applied to field annunciation testing [except we call it end-to-end testing], where each alarm point is tested one at a time to confirm its functionality, something that our company's management also seems to no longer be willing to commit the resources to, incidentally...

But racookpe1978 speaks of "tracing cables"...

What exactly do you understand 'point to point testing' to mean?

CR

"As iron sharpens iron, so one person sharpens another." [Proverbs 27:17, NIV]

 

[waross]

racookpe1978;
I agree with you on the futility of following cables. Antone who suggests it as a safety measure is displaying a complete and dangerous lack of field experience.
Sometimes a cables is "walked" as a last resort to try to locate which field device the cable feeds. Any one who has done this in a large plant knows how difficult and misleading this method is, not to say time consuming. After following a cable for a couple of hundred feet to a field device, a continuity check often fails. Somewhere you have lost the cable on a corner or in a bundle and have been following the wrong cable.
By Point to Point testing I mean a continuity test from end to end of the circuit. Two workers with communication. On the command of one worker, the second worker at the other end of the circuit makes and breaks continuity, usually three times. Some conversation is encouraged to make the intervals between make and break and repeat more random.
This is the basic test, there are refinements to deal with special cases.

Just checking for voltage is a must before the point to point test, but does not prove that the correct circuit is locked out.
I worked in one plant where the prints were not dependable. There were hundreds of electric heat trace circuits, all on thermostatic control. It was fairly common for workers to turn off the circuit indicated by the prints and then find that the field junction box was still energized. Bad.
Sometimes the field junction box was dead, but in a short time the thermostat would cycle on and the junction box that may have been assumed to be dead was now hot. WORSE.
If you have voltage, you can prove the circuit by switching the breaker or control on and off several times. Still two men with radios and a meter.
I have been involved in a few trouble shooting sessions assisting in tracing out circuits in order to do a safe and proven safe lockout when the prints were in error.


Bill
--------------------
"Why not the best?"
Jimmy Carter

 

[Parchie]

I maybe late in chiming-in but don't you people use ground clamps prior to working a LOTO-approved task?
Just my two cents.

 

[Skogsgurra]

Very seldom seen it done in practice. And never by mechanical guys that are doing adjustments and repair.

Safety is not only for electricians. And, if you have a high powered DOL motor - there will be enough fireworks to cause injuries when the clamp is applied.

Gunnar Englund

http://www.gke.org

--------------------------------------
Half full - Half empty? I don't mind. It's what in it that counts.

 

[Parchie]

Very seldom seen it done in practice. And never by mechanical guys that are doing adjustments and repair.

IIRC, mechanical guys use blocking and releasing of stored energy as part of the LOTO in the same manner electricians use grounding and chaining to compliment the installation of blinds, restraining barriers, etc. Bluntly, a fool-proof LOTO could help a lot.

 

[waross]

My issue is the lack of verification that the correct switch has been locked out.

Re Chaining;
Chaining is commonly used here for higher voltages. Generally above 4160 Volts. When there is a possibility of a circuit being energized from more than one source grounding jumpers or "Ground chains" are used for added protection.
At 480 volts and 600 Volts ground chains are never used.


Bill
--------------------
"Why not the best?"
Jimmy Carter

 

[MintJulep]

Safety and Software do not and can not ever go together.

No amount of standards and documentation purporting Safety Integrity Level can change that.

This is a perfect example. Because all it took to violate the SIL was for the on-site commissioning engineer or a plant engineer to hack some of the code out.

Where is that documented in the SIL certification process?