VFD motor control causing soft error in microcontroller flash memory

[elarson]

The basic setup is a microcontroller that is providing a 0 - 10 DC voltage level to the VFD. The VFD is connected to a 120VAC fan motor (small horsepower). The problem that is occuring is the flash memory in the microcontroller is getting corrupted. Reprogramming the device corrects the corruption problem.

I would like to be able to recreate the problem in the shop, but haven't had any luck. How can I corrupt the flash memory? I haven't been able to see any frequency in the field that might cause the problem. As anyone else experienced this type of problem?

 

[cbarn24050]

What micro are you useing?

 

[elarson]

The micro is a philips P89C51RC

 

[jraef]

How is the VFD connected to the motor? Shielded cable or wire in steel conduit? If you used portable cord (like extension cord material) you have made a somewhat powerful local FM transmitter.

Eng-Tips: Help for your job, not for your homework Read faq731-376

 

[elarson]

The wire connection is portable cord. There is actually two microcontrollers in the system. The local board receives a command from a station 10ft away(the second micro which is being corrupted). There is 500 hundred of these systems running. And at least 1 fails a day. Failure meaning that the station device has to be reprogrammed. Already tried a few filters with out success. The property is 1600 miles away and I haven't been able to recreate the problem in the lab. If I had some clue as to what it takes to corrupt the micro, I would be able to filter the problem.

 

[itsmoked]

In no shape, way, or manner can I imagine any kind of noise EVER corrupting flash memory!!!

Several questions:

Is the flash capable of being "self modified" by the CPU?

If it is, you would be more likely,(like a million times),to be having a software bug that is commanding a bad self modification.

Or a soft error that causes the same thing to happen.

Otherwise your boards must be experiencing a RAD of ionizing radiation that is in the process of frying the entire processor.

Keith Cress
Flamin Systems, Inc.-

http://www.flaminsystems.com

 

[Skogsgurra]

Why do you try to reproduce the problem on your bench?

That's rather futile.

Instead. Get your scope and HF probes (isolated) as well as a Fischer HF clamp and stick it in your car. Also, get loads of capacitors, chokes, ferrites, hose-clamps, filters, shielded cables, EMC glands, braids, bolts, washers and nuts and tools and whatever you need to work in the field.

Kiss bye to wife and children and go there. Be prepared to stay for a week - at least. If you have an installation with 500 units that fail once a day - you have a serious problem. That is nothing you can solve in your lab.

I would go to management and tell them that this is something they have to solve once and for all. Not a piecemeal job.

Gunnar Englund

http://www.gke.org

 

[elarson]

Kieth and Gunnar, thanks for your reply.

To answer your questions:

The flash has a boot rom which can allow in application programming. I'm going to bring this up again with the CTO.

Our station controller was basically a 3 speed fan controller. It pulled on a different relay for the different windings in the motor. This had worked fine for years. When we installed a VFD to allow quieter fan operation we started seeing the devices memory getting altered. The station controller has been installed all over the world for many years and in the thousands without a problem. Of course, they are not being used as VFD controllers.

I would like to rule out the fact that the flash memory is being directly altered by EMI if possible. I haven't been able to find very much information on the subject.

If the problem would occur on a more regular basis at an individual station at the site, it would be easier to monitor the problem. It is too random for anyone to stay there waiting for it to happen.

 

[Skogsgurra]

elarson,

I agree that the memory is not changed directly by any EMI. It is most probably something that happens either via the power supply or via the reset circuit. Another possibility is that the address gets corrupted so that the processor goes to never, never land and that something happens on the way there.

Is changing the flash memory the only thing that happens? Or do you have other problems like frequent restarts or changes of operation modes? Do you have to restart the equipment often?

I did not mean that you should go on site and try to catch the thing when it happens. I mean that you should check things like interference on power supply lines, ground potential, reset circuit and such things. If you find interference there when the VFD is running it is a good starting point. Make modifications while monitoring interference levels. You will hopefully find the main road that the interference is taking and also a way of blocking it. Then, there may be another, less important, road that also needs blocking. But usually, there is just one thing that needs to be done to make the equipment more reliable.

Good luck! And please leave some feed-back on how things are going.

BTW: Did I misunderstand the number of units in one place? Are there 500 of them? And about one of them being disturbed every day? That is, as I see it, a serious situation.

Gunnar Englund

http://www.gke.org

 

[itsmoked]

elarson; You haven't found much on flash being altered by noise etc. because it as I said it is unlikely! The required energy and voltages and charging times are not something provided by "noise". Self modifying code is something that if allowed, can then be caused by a crashed cpu. There are several ways to prevent this both in code and in hardware. Very often designers do not follow 'industrial quality' programming rules. These rules set up byzantine rules for allowing self programming and detailed error checking and correction. The same designer who does not put into effect these 'industrial grade reliability' practices may also miss some hardware details that then make the system noise susceptible.

These tricks and details are learned by most through diligence and the school of hard knocks. I was in on a project that put the first microcontrollers into rail cars to control refrigeration. Having some of the problems like the one you're describing on a fleet of 300ton boxes that are uniformly distributed across the country taught me some very hard lessons.

I would suggest you have someone go over the design if you are still making them.

Otherwise you are going to have to shotgun solutions on all of them which may be a huge expensive headache.

Are you talking about a bunch of units here and there that all worked fine until each one was hooked to a VFD or one VFD gets added to a site and all the units start being victimized?

What kind of processor are we talking about?

Keith Cress
Flamin Systems, Inc.-

http://www.flaminsystems.com

 

[DickDV]

Keith, just a friendly sidenote here---those would be 286,000lb cars or 143 tons, don't you think?

I know railcars are heavy and getting heavier but not 300 tons!!!!!

 

[davidbeach]

No, not 300 tons. Since Keith has mentioned work on rail passenger cars, I think he just added a zero and really meant 30 tons.

 

[itsmoked]

Yeah some serious rounding error there. Was thinking high 200 hundred thousands, about 300,000pounds => 300T. That's what I get for conversing with intelligent knowledgeable people!

Keith Cress
Flamin Systems, Inc.-

http://www.flaminsystems.com

 

[elarson]

The micro is a philips P89C51RC.In-application programming instructions are activated by a BOOTROM area. There is also a security bit that can be set to keep from erasing the flash. It seems to me that it would be more likely that RAM is being corrupted causing a block erasure. I am also certain that our ICP is not setting the security bits. Our ICP also doesn't have the capability of reading back the memory. I think I'm going to suggest an after market ICP so we can figure out how much of the memory is getting corrupted. Is it a bit, byte or a block? It might help to lead to some conclusions.

The whole properties of 500 stations each have their own VFD controller. All the stations worked fine before the installation of the VFDs.

A number of people from the team have been out to the site to monitor the devices. We’ve designed 2 different filters without success. A common mode disturbance has been ruled out. The latest tests we are running are for brown-out conditions.

Thanks again for the support.

 

[itsmoked]

That IS what's happening. If it can it IS!
The very first problem that you get when you hammer a micro is a reboot. Next would be the occasional data bit flip. next would probably be a changed register value.

Let me suggest you get another designer to look at your system with a fresh eye.

Please keep us informed elarson.

Good luck!




Keith Cress
Flamin Systems, Inc.-

http://www.flaminsystems.com

 

[Skogsgurra]

The solution may be dependent on where in the world you are. And it is not at all self evident that you or your company shall better your equipment.

If you are in Europe, there are Euronorms (EN 61800-3 for instance) that say how much EMI a VFD is allowed to spit out. It is obvious that the VFDs in question are emitting a lot more than allowed - or that your design is so lousy that it cannot take any EMI at all. I rule out the latter.

If you were in Europe, you would have an EMC consultant measure the conducted emission and when you have the numbers - they are probably way above what is allowed, they usually are - then you demand that the VFD supplier/installer do something to reduce EMI. A mains filter is usually what is needed.

What make are the VFDs? What filters have been installed? Is there any EMC declaration?

Gunnar Englund

http://www.gke.org

 

[Marke]

Hello elarson

It is highly unlikely that the EMC type interference would be able to directly alter the memory contents.I think that you can discount that possibility.
If the firmware is being altered, then it is most surely being done by the processor itself.

The most common cause of this, is poor pcb design and layout, particulary around the rest input.
With the 8051 family, you must hold the rest line for at least 12 cycles to be sure that you have fully reset the processor. If you reset for less than this, you will reset part of the chip but not all of it and the processor can disappear into it's own version of the firmware. This could enable the bootstrap or part of it.
I have a few suggestions that need to be looked at.

1. Use a fast special purpose micromonitor to control the rest input to the micro. This should have an accurate voltage threshold that is appropriate for the micro so that it causes a reset if the supply voltage begins to drop, well before the uC gets into an undetermined state. - DO NOT rely on the old RC rest circuit!

2. Use a good continuous ground plane under and around the uC

3. Apply good chip type bypass capacitors from the supply pins to the ground plane.

4. Position the micromonitor as close to the uC as possible and bypass well to the ground plane.

4. Use an earthed shield between the primary and secondary windings on the transformer supplying the uC, or use a cetre tapped secondary with the center tap connected to the uC ground plane. A lot of the HF noise can be cpacitively couple through the supply transformer and once in the circuit, it is hard to get rid of.

Best regards,


Mark Empson

http://www.lmphotonics.com

 

[cbarn24050]

EMI can corrupt flash, at least in a PIC16F870. The MLCR trace from the chip to an edge connector, about 3" long, seemed to be picking up something and putting the chip into "prgramming mode". I had to cut the track and hard wire the pin to 5v to cure the problem.

 

[itsmoked]

cbarn; Do you have the recommended 10K pull up and the 100 ohm series resistor on the MLCR pin?

Keith Cress
Flamin Systems, Inc.-

http://www.flaminsystems.com

 

[cbarn24050]

Hi Itsmoked, of course.