Damaged Yaskawa VFD is being blamed on me

[skbutler]

I installed two new identical Yaskawa VFD Model CIMR-P7U47P51B-107 to drive new identical 10 hp 480Vac 3ph motors. This was as simple an application of VFDs as you get. No network control, nothing unusual. The only problem I had was terminating the signal wires that connected the two units so they could communicate with each other. The wires I used, at the factory's rep's suggestion was a short piece of Belden cable that came with the pressure sensor. The wire was 18 awg stranded. Two of the terminals required nesting two wires in the terminal. Due to the very small size of the terminals and the angle I had to work, it was next to impossible to fit 2 wire into one terminal and make it look professional. After I had completed the installation and had both units running perfectly, I decided to go back and replace the Belden cable with stranded wire with Belden cable with solid wires. PLEASE no need to comment on this next step. I took both units to local and turned unit B off but did not de-energize the VFD. (I know, I know.) My hands were far enough away from a hot terminal that I wasn't worried. I removed the stranded wires and was in he process of inserting the low voltage solid wires when BOOM. I got slammed backward. I was out for maybe 30 seconds. The VFD was fried. The three 60 amp fuses were all blown. After removing the VFD and replacing with a new unit I photographed all terminals and all external metal surfaces to show a complete lack of any arc mark and shipped the unit back to Yaskawa. They sent back a two page report (form style) that concluded "IGBT blown gate drive board and the DC bus capacitors damaged. Input phase R, S, T open at + side. S, T open at -side. Damage to the unit indicates that an external source caused the damage to the IGBT." Since they have the unit I can't verify any of their findings. But I know I didn't do anything to cause the unit to blow. I am a boiler expert, not a VFD expert even though I'm an EE. If an "external source" caused the damage, shouldn't there be some external physical indicator? Opinions would be welcome.

 

[itsmoked]

I believe you caused this and here's why.

If you where "out" for even a second there is a better than even chance you do not, and will not, ever recall completely or correctly the few seconds before the event.

In the second before the event I believe one of your wires (especially since they were solid) poked somewhere it wasn't supposed to. Most likely through some gap around the terminal blocks and touched somewhere on the controller. That could easily gate ON the IGBT in a destructive uncontrolled manner. It could well have been one of the wires you weren't working specifically with in the cable.

The aforementioned description would leave no exterior physical damage at all.

Keith Cress
kcress -

http://www.flaminsystems.com

 

[rasevskii]

for skbutler:

You were in fact very lucky to have not gotten killed or disabled for the rest of your life. The rules are always to totally isolate and insure that equipment is actually de-engergized before doing any work on it. Also never work alone. Another person has to be available nearby just in case.

Possibly some bosses were pushing the situation or the equipment could not be shut down. Then just refuse to work on it.

I have seen this all before and have had a few close calls in my previous work in the field.

rasevskii

 

[LionelHutz]

Yes, you most likely caused the failure.

VFD's can be especially dangerous to work on due to the capacitors storing energy. I even recommend safety glasses and hearing protection when just working near them (while enclosures are all properly closed).

 

[jraef]

Such are the dangers of working hot on electronic power conversion equipment.

Without crystal balls that have time travel capability, there is no definitive way to determine exactly what went wrong. But I think itsmoked has the best likely scenario. In the "olden days" a signal wire would have been going to a dedicated signal processing daughter board that would have likely been isolated from the firing circuit, so the kind of damage you experienced would have been more difficult to imagine. But modernization / cost cutting / down sizing of VFDs has led to a lot of SMT single or combined board systems and hence a much higher likelihood of a tiny error in touching a board trace causing catastrophic failure. Could have been the wire you were installing making contact with a board trace that affected the transistor gating, could have been a strand off of the wire you removed.

The only person I have ever known killed by a drive was a VERY experienced technician who worked hot on a large DC drive and touched the tip of his screwdriver to the wrong place (or so the witnesses said). The IGBTs exploded, he jerked back from the flash and reached out to catch himself, grabbing the DC bus. Even though the fuses had already cleared at that point the residual energy in the caps put him in cardiac arrest.

Don't do that again, you were lucky if you get away with only a financial penalty...


"If I had eight hours to chop down a tree, I'd spend six sharpening my axe." -- Abraham Lincoln
For the best use of Eng-Tips, please click here -> faq731-376

 

[skbutler]

"PLEASE no need to comment on this next step" was an effort on my part to inform everyone that I fully understood how foolish it is to work inside a hot panel, VFD or not, 120 v or 480 v. I get it. I worked on radars in the military that operated at over 50kv. After de-energizing the system you had to use a 5 foot glass grounding rod to touch certain part to bleed the residual energy to ground. I get it.

The factory tech says that "Damage to the unit indicates that an external source caused the damage to the IGBT." What does "an external source" mean? I think that shorting the signal wires together or to ground will not blow the IGBTs. Am I wrong? I've seen sensors damaged and submerged and not damage anything. But like I said, I'm not a VFD expert.

I appreciate itsmoked's comments but I didn't loose my short term memory, which can happen with an electrical shock and usually does occur to one who has been Tasered. The boom occurred as I drew my hand after I had terminated the last of the two solid wires. The wires were only 1.5 inches long and U-shaped. The terminal strip is monolithic and it is not possible to reach a grounding surface with either end of the wire terminated. The stranded cable was not in the cabinet when I installed the solid wires. There were no dangling wires. I'm going to try an upload a pic of the wires and what I have described might be more clear. But if itsmoked is correct, that if you wanted to produce identical damage to an identical VFD unit you could short out certain terminals on the terminal strip or short one to ground and the VFD would blow without producing an arc mark at either end if the offending wire, then I can go no farther without examining the subject VFD.

Other opinions still welcome but please no lectures on electrical safety. I promise I'll never do it again.

 

[Skogsgurra]

Sorry. You have to face it. Fair or unfair. We are probably not going to give you much support in this. Guesses are guesses.

You know: "He who touched it last done it"

Not much to do. Just accept it.

Gunnar Englund

http://www.gke.org

--------------------------------------
100 % recycled posting: Electrons, ideas, finger-tips have been used over and over again...

 

[skbutler]

Skogsgurra, you are probably right but guesses are not what I was looking for. The front end of the drive was pristine, even under a 10x ocular. If 'itsmoked' is correct in that it is possible to short certain front end terminals together or a terminal to ground and cause the damage found in this case and NOT leave a tell-tale mark, even under magnification, then I have no complaints.

I do beta testing on an operating system for a major computer company (NDA prevents me saying which) and on one release they were having monitor issues. Some monitors would display weird colors and blink on and off. While testing this release my monitor died. I've used computers since 1975 and have had one monitor die, the monitor I was using testing a computer operating system that was known to cause weird monitor problem. I complained. That was just too big of a coincidence for me. I think most people would have added 2 and 2, just like I did. The computer company said there was no way the OS would cause that particular problem and had me take the monitor to a repair shop, which they paid for. A wire under a metal fastener had shorted to ground. The OS had nothing to do with the monitor dying. Some times coincidences do happen... that's why they call them coincidences.

 

[FreddyNurk]

'External Source' in this case means that something external to the drive caused the issue, at least as far as I can interpret. That doesn't exclude whatever you were poking it with.

Skogs is right, theres no other way around it.

 

[jraef]

I think you are interpreting the "external source" to be associated with the power source to the front-end of the drive. Not necessarily so. If you read what itsmoked said (the irony of his handle just struck me), even a seemingly casual contact with a board trace or any other PCB contact point could have inadvertently caused one or more of the IGBTs to fire at the wrong moment, creating a dI/dt or dV/dt event that cascaded into the full blown failure in a matter of milliseconds. For that matter, it may have even been a static discharge onto the board from your finger and it jumped further than you think it would have. We don't have any way to know now.

But from a forensic standpoint, what Yaskawa is saying is, it wasn't the result of a failure of one of their components, therefore it was "external".

Count your lucky stars, kiss your wife and kids, chalk it up to experience and move on.

Oh, and buy a lottery ticket... you are one LUCKY dude!

"If I had eight hours to chop down a tree, I'd spend six sharpening my axe." -- Abraham Lincoln
For the best use of Eng-Tips, please click here -> faq731-376

 

[skbutler]

Good post jraef
Initially I did assume that external source was an electrical service of some kind but then I realized it could be anything.

The only thing I disagree with you on is, "But from a forensic standpoint, what Yaskawa is saying is, it wasn't the result of a failure of one of their components, therefore it was "external"." What happened is this. The damaged unit was received by Yaskawa and placed on the returned goods shelve. After a day or two a tech took the drive apart, He checked a few components and took a few pictures. He filled out the REPORT form, 2 pages of boxes to check, ie, Good, no good, NDA. He didn't see a smoking gun. The entire "forensic' report was two sentences: "IGBT blown gate drive board and the DC bus capacitors damaged. Input phase R, S, T open at + side. S, T open at -side. Damage to the unit indicates that an external source caused the damage to the IGBT.

Has a IGBT or a gate drive board ever failed due to an internal failure? If such a failure has happened, did the failure cause other devices to catastrophically fail? If no such failures have ever happened then it is unlikely that it occurred in this case.

A professor of mine started his class on Root Cause Analysis with "let me tell you a little secrete about Root Cause Analysis in industrial accidents. You never ever ever ever find the root cause in anything except the simplest accidents. You narrow the possible causes down to three or four... if you are lucky. You pick one to call the root cause and then you re-engineer it and the other two or three"

Anyway, thanks for the input.

 

[jraef]

I don't necessarily disagree with you that "stuff happens", but lets look at the big picture. Component failures can happen, but they are relatively rare. Severe spikes can cause damage like that, but they too are very rare. With all the billions of unit-hours that VFDs are connected to power sources to where they could fail like this on their own with no influence, the incidence rate of it actually happening is extremely low. Take that low low overall incidence rate and factor in how often it would happen at the exact moment that someone was standing in front of it with the door open and connecting a control wire, and you have a likelihood that is so infinitesimal that it's very difficult to accept. The likelihood that you were in some bizarre unfortunate way a root cause is far greater. Occams Razor...

"If I had eight hours to chop down a tree, I'd spend six sharpening my axe." -- Abraham Lincoln
For the best use of Eng-Tips, please click here -> faq731-376

 

[waross]

Well you were connecting the wire that let the two units talk to each other. Without going through the proper boot sequence, but being connected on the fly, one unit may have started talking when it should have been listening. That could have initiated a transient that triggered an IGBT when it shouldn't be triggered. You only need one IGBT to turn on at the wrong time for all sorts of bad stuff to happen. Like lemmings, other IGBTs may join the first unfortunate on the journey to the big trash can in the sky.
Most techs won't find or provide a definitive answer to this possibility. You would probably need a senior designer and his charge-out cost for the time may be a lot more than the value of the drive.



Bill
--------------------
"Why not the best?"
Jimmy Carter

 

[OperaHouse]

I could believe this is a software issue and have seen this happen in products now and then. Software and hardware is often not designed for all possible cases. Think of a hardware reset either by an electrical discharge or communication issue in the middle of normal operation. Think of this real example that happened years ago.

I was about to leave a parking lot in my 97 Ford Explorer and was stopped at the edge of a sidewalk waiting to enter traffic. Normal pressure was on the brake when all at once the engine raced to 2,000 rpm and the vehicle lurched forward. I thought my foot had slipped. This happened two more times while at stoplights and I pulled off to assess the problem. Finding nothing I started the vehicle and the problem became obvious. It just happened that I was charging a gel cell from my cigarette lighter and the low ohm resistor to limit current started burning. The battery clamp on the car battery had cracked.

It was this poor connection that had caused the car computer to reset by a spike or low voltage. That caused the electronically controlled Idle Air Control valve to fully open and race the engine. Prior to engine racing there was never any stumble of the engine. The alternator and the extra gel cell battery should have kept the computer alive. Still, a little code that would have checked to see if the motor was spinning would have prevented this from happening. No one anticipated this case would have happened. No cop would have believed my story in an accident.

So, this makes it their fault? Not really. You were performing an operation in a mode that was not authorized.

 

[sibeen]

Most techs won't find or provide a definitive answer to this possibility. You would probably need a senior designer and his charge-out cost for the time may be a lot more than the value of the drive.[/d]


I'd have to say that there conclusion would be no more than a 'best guess', basically the same as the techs. Unless a problem has occurred on more than a few occasions, and starting to effect the business model, no manufacturer is going to invest much time in examining the root cause.

I had a case(s) quite a few years ago where the output filter caps of some fairly large power conversion equipment were failing in a rather catastrophic manner. I was the technical manager for my region, Oceania, and could not convince the factory management and design engineers, who were located in mainland Europe, that there was a major issue. They continued to fob off any suggestion of a problem, whilst I was pulling my hair out having to deal with irate clients.


It wasn't until I was speaking to my counterpart in the UK, on a separate issue, that the underlying issue came to light. They were experiencing the same faults and we jointly issued a report detailing the failures, and made sure that the senior management of the company were included in the correspondence. Once the senior management was made aware of the problem, and the associated customer issues, the design team 'flew' into action.

The result, do what I, and UK counterpart, had been screaming at them to do for 18 months, change the manufacturer of the capacitors.

 

[ischgl99]

You are fighting a loosing battle on this one. There is the remote possibility there is an issue with the drive that caused the failure, but based on what you have described, you are most likely the cause of the failure and you will never get anyone to investigate this further unless you are willing to pay them for it.

By the way, if you worked for my company, you would have been fired for failing to us proper lock out / tag out procedures while working on the drive as well as not using arc flash protection on an energized drive.

 

[Skogsgurra]

I wouldn't be that harsh. Such declarations often reflect a degree of wishfull thinking.

Gunnar Englund

http://www.gke.org

--------------------------------------
100 % recycled posting: Electrons, ideas, finger-tips have been used over and over again...

 

[OperaHouse]

It was 3am and I was tired. Don't think my train of thought was easy to follow so here it is again.

A three phase VFD has six switches in the output. Two or three of these switches are on at any one time. These can represent a short to the DC supply through the motor. These are switched on and off 5,000 or more times a second. The inductance of the motor resists initial current flow and allows the current to ramp up. The switches are turned off in time by design and monitoring of the current in normal operation.

In a natural failure one of these output switches shorts. If the microprocessor is awake, it senses a high current draw and shuts down the drive so there is no massive failure of all the output devices. This is a change from the old days when everything was wiped out. This common type of failure would seem like a warranty to the tech.

In an over voltage or surge like lightning the microprocessor has no control and everything shorts. This takes out several of the output devices and the power supply. Everything burns up till the device disconnects from the line. This is generally considered the customers responsibility. Pretty simple decision process.

The VFD microprocessor has power monitor chips that look at the mains supply and and power for the processor. This allows safely controlled start up and shut down under software control. But what if the microprocessor went into a reset, halt, or got stuck in a loop in the middle of a cycle. Those outputs that were turned on would stay on. Massive over current wouldn't take long to destroy several output devices.
This would look like an over voltage failure and the customers responsibility. Control boards generally have a connector where the factory tech can power up the microprocessor. If the tech powered that up and it still functioned, the blame would be on something external and not a control problem.

A zap could have reset the computer or a misinterpreted communication could have got the microprocessor lost. I've designed small control systems. At a place I used to work we had this old probe that I swear was a cattle prod. It would throw down arcs to all the terminals connected to the outside world. This test didn't meet any code requirement but if it passed the design was pretty robust. This doesn't relate to this issue but small microprocessors with internal clocks are magnitudes more stable than those with crystals. I was testing one of these and zapping it every which way. Data was streaming out to my video terminal and couldn't get to fail. Honestly, after passing basic tests I will keep pushing my luck till there is a total failure just for fun. All I managed to do was destroy the video terminal. That had an opto isolated input and was supposed to to be immune from that. It was also my own terminal that I had brought into work. You can never predict what happens with spikes!

 

[waross]

Hi Skoggs.
Was your comment directed at ischgl99's post?

By the way, if you worked for my company, you would have been fired for failing to us proper lock out / tag out procedures while working on the drive

The last 5 industrial site that I have worked at over the last 4 years had a "Zero Tolerance" policy for violations of the "LOTO" (Lock Out, Tag Out) rules. The penalty for violation is immediate termination.
Arc flash is a "work in progress". But when the rules state that arc flash protection must be used, violation of the rules again will be met with immediate termination. There may often be discipline for the worker's immediate supervisor also.
At one plant the training center was outside the main security gate. We spent four days in training/orientation before we were allowed to pass the main gate into the plant. One day was spent on LOTO procedures.
In more and more of the plants, when arc flash procedures must be followed, the worker must carry a card attesting to the successful completion of a recognized arc flash training course.
I am expecting to hear sometime this week when I will be scheduled for two days of recognized arc flash training.
An anecdote:
I was aware of a small group of supervisors (owners reps and contractors supervision) in a large plant who were adept at avoiding proper LOTO procedures.
One day there were two serious LOTO violations that could not be covered up nor could scapegoats be set up in time.
This was a very lucrative cost plus contract. The contract was terminated and a new contractor was on site the next day.
The downtime on this rebuild was reportedly One Million dollars a day. Despite that, in a jurisdiction where the CEO of a company may be held personally responsible and may be sent to jail in extreme cases of workplace injury, A violation of the safety rules IS NOT TOLERATED.


Bill
--------------------
"Why not the best?"
Jimmy Carter

 

[Skogsgurra]

Yes, Bill.

I wouldn't fire a guy that seems to be qualified to do a job and that did a mistake.

By wishful thinking, I mean that we often wish that we lived in a perfect world. We don't. And I accept that.

Gunnar Englund

http://www.gke.org

--------------------------------------
100 % recycled posting: Electrons, ideas, finger-tips have been used over and over again...