how to manage program changes with many programmers

[FTSSTL]

Our plant has over 25 Allen-Bradley processors, including ControLogix 5000, PLC-5, SLC-5, and MicroLogix. When I arrived about 3 months ago, the method for saving programs was very chaotic. Now I am trying to bring order to the method for saving our programs and I can use some advice.

We currently have a rule that when someone makes a program change, we require them to make a screenprint of the changes (showing the "old" logic and the "new" changes) and then put the paper copy of the screenprint in a big logbook. The intent is that if something in the plant is not working, our programmers and maintenance poeple can check the logbook for any programming changes.

Problem is that the logbook quickly becomes large and hard to use, and many people who make the changes to the PLC program are fighting an emergency and feel that they do not have time to record their changes and so they fail to record their changes.

We have considered RSMACC, but my understanding is that it will identify a person by his login name if they do an online edit, but it will NOT identify the actual edit itself, which would not help in troubleshooting.

I am considering adopting a policy like this:

[1] all changes to PLC code should be written with an identifiier test bit (something like "TEST_0001" which MUST be included on the rungs where the changes have been made. If changes are required in several processors for the same issue, then the same test bit should be used in all the processors.

[2] the changes should be written in a manner that allows for them to be cancelled simply by RESETTING the test bit, which would then return the logic to the state that it was in before any changes were made.

[3] the log of test bits would be maintained as a master document, so that if something is broken, then the maintenance prople can consult the master logbook to see if someone had made a logic change that may have affected whatever is now broken.

I think that this may be a partial solution to our problems, but I am worried about a couple things:

[1] it takes extra code to make changes so that the logic reverts to its original state when a test bit is RESET, and by requiring extra code, there is a greater chance for making a mistake

[2] someone could still simply make code changes and omit the test bit; in other words, it is not really enforceable except by agreement of all the parties involved.

I would appreciate any and all comments and feedback about this, as I am in the initial stages of formulating a system. What other systems are out there?

Help please!!

 

[jde777]

I would suggest looking into RSMACC and RSAsset Security.
With rsmacc you can assign users and permissions to workers.

http://www.rockwellautomation.com/rockwellsoftware/assetmgmt/rsmacc/

Regards
Justin

 

[jde777]

BTW RSMACC will identify program changes utilizing verification, events and audits

 

[FTSSTL]

Thanks for the advice jde777.

It is important for any of our plant personnel to be able to access any changes in the PLC code very quickly. Do you think that RSMACC will give them immediate information about what has been changed, where the changes are located (file # and rung #), and how to cancel the changes to get back to the original logic? Or will they have to do some digging to get that kind of information?

 

[jde777]

RSMACC will run on win 2003 server. A master file, i.e. acd or rss, can reside on this server and any change to the file will be audited and logged against the master. The log will contain what was changed and what user changed it. In addition MACC can be configured such that the user making the edits will be prompted to make a note as to what changes he/she is making and why. This info will be kept on the server. The info you are wanting is available immediately for evaluation by the administrator of RSMACC.

Hope this helps
Justin

 

[dV8r]

As an alternative you may want to look into the GE Fanuc Proficy Change Management package. It does everything you mentioned, even for AB PLCs, many other PLCs, MS Office files, ACAD files, and many others.

http://www.gefanuc.com

 

[TWControls]

dV8r - I couldn't find this on GE Fanuc's site. Can you give a more specific link?

TW Controls -

http://www.twcontrols.com

 

[dV8r]

http://www.gefanuc.com/en/ProductServices/AutomationSoftware/Asset_Mgt/ChangeMgt/index.html

 

[BAnderson]

I ran into a similar problem here is how I went about resolving the problem. We are a larger manufacture with many sites. We have six on site developers and 5 support personal that need to be able to trouble shoot problems and reload PLCs.

We use a program called SubVersion to back up and track previous version of our plc programs. As a side note we also keep our electrical drawings and other documentation in here. In the beginning comments of the plc program we keep a log of what we changed on what date.

We also use a CMMS (Computerized maintenance management software) and enter the changes we made to a particular piece of equipment. This software was written in house because we wanted some features we could not find in other packages. But there are many out there.

CMMS Packages
MP2

http://www.datastream.net/english/products/mp2.aspx

iHeat

http://www.frontrange.com/ProductsSolutions/Detail.aspx?id=53

Source Control Packages
SubVersion (Free)

http://subversion.tigris.org/

CVS (Free)

http://www.nongnu.org/cvs/

SourceSafe

http://msdn2.microsoft.com/en-us/vstudio/aa718670.aspx

 

[Kiljoy]

I know this may be a bit simplistic, but If they can't bother to hit the print screen button, they may not bother with the other suggestions.
When we make changes, we simply save the original on the server and then "save as" a revision letter. E.G. MainProgram1.Rss now becomes MainProgram1a, b, etc. Then we open the EDIT>Revision Notes and put in a blurb about the changes. If the new program doesn't work, simply download the last saved rev. You can open multiple instances of logix and compare the two programs as need be.

 

[jonesy29847]

In one of our plants we have a policy of testing edits for a couple days before assembly, if a problem arrises the tech can untest the edits to see if the changes are the cause, the caveat is that to make changes the edits have to be untested and the new changes become part of the test.

 

[eeinpa]

I come from an embedded controls hardware/firmware environment, not a PLC environment, so my perspective might be a little different. The solution for PC and embedded software (pretty much across the board) is to use "version control software", as another user mentioned. The last product I used was called PVCS.

These systems allow multiple users to check out and change files, have merge tools allowing you to combine changes from multiple files, and allow you to create multiple versions and branches of a "tree" of files. Probably the biggest single personal advantage is being able to go "oh, drat, i wish i hadn't made those changes yesterday" and being able to go back and get the earlier version. My last employer debated a long time over the "high cost" of this software. It probably paid for itself the first day. If you are doing multi-user software development, you MUST have it!

The versions I have worked with are geared towards software source code, but most of them can cope with binary and ASCII files, with features varying accordingly. I suspect one of the companies making version control software must deal with ladder logic.

The drawback is that these packages are pretty complex. You often need one person to act as administrator to really understand what's going on with them.

It's probably going to require some research, but if your manager has any clue, they will see this is completely necessary and support you 100%. IF!

Good luck!

 

[TheMcNasty]

SubVersion is definitely a good solution to this. Not only do you track changes, it makes them reversible and keeps the files in a central location so there are no local copies that end up radically different from the main copy resulting in a combination nightmare. Speaking of that, subversion has a combining tool that will show you what is different and let you chose what parts to combine or leave out.

Oh, and it's free.

 

[richs]

Hiya-

I use CVS, but then as mentioned, this is primarily an ASCII version control. It does not do well with binary. One can put a binary file in a "wrapper", but it does nothing to show the changes. Ladder logic, being a GUI programming language doesn't really lend itself well to this.

Does subversion (the "newer" approach to versioning software) allow for this? Or is like CVS looking for differences primarily with ASCII (or text) files?

Cheers,

Rich S.

 

[modhauler]

We have about the same situation at our facility; LOTS of A-B controllers, about 30 OIT's running different WonderWare 9.0 packages , 29 maintenance technicians, BUT ONLY 2 programmers! We had great discussion about allowing all of the technicians unlimited access to the actual PLC code, and it was determined , for SAFETY reasons (personnel AND process), not to do that. We provide copies of RSLadder running so they can go online in real-time and see whats happening in code, but cannot implement any changes. For the most part, that satisfies their needs. We have found, that except for process changes, equipment addition/deletion/modifications, that the code itself, once written, de-bugged and commissioned, usually didn't require a lot of future changes.
Level presets, timer delay values, QC equipment parameters, temperature and speed presets, and the like DO though, and we build this function to adjust them into the Wonder MMI screens.
It is a policy between myself and the other programmer to document every change in some manner; usually in the rung or function block comment area. We also make is a set routine to back-up the plc code 1.) after every PERMANENT change, (not test edits), or 2.) on a weekly basis as to capture dynamic quality control equipment settings. With that being said, it IS still a "work-around", not the structured automatic tracking/back-up scheme I would like. We looked at RSGuardian, but it was somewhat clunky and pricey.

What would be really nice is if Rockwell would make the "difference detector" run automatically whenever edits are assembled, force the person to enter a comment about the change (kinda like the Windows Server 2003 shutdown dialog comment box) and store it in a usable, printable format.

I am going to check out SubVersion as well , as it sounds like a possible step in the right direction towards "Electronic Change Management" for us too.

I did not intend to hijack your thread, just state you aren't alone in this situation.

Keep us posted on your progress.. John H

 

[spook6869]

I had some thoughts...

I personally use your TEST0001 bit methodology, and it works well for me. I copy the code onto a new branch on the same rung. I use an XIO to disable the old code, and an XIC to enable the new code. Forcing the bit turns enables the old code. The second advantage here is it makes it easy to find the location of the modified code later on.

For very large changes in your RSLogix 5000 controllers, you could store earlier versions of code in the unscheduled programs folder and not need to mark it at all. Reverting back to the earlier program is cut-and-paste.

...However, I agree with mod4 on a very fundamental level. Obviously I know nothing about your plant. Generally speaking, however, unless you are adding features to your equipment or improving your process, changes to the compiled code should be very rare. Dynamic values should be in containers, and are optimally accessible from the HMI. The prinicple "emergency" that should justify a program chaing is the discovery of a bug. If this happens often, it might be the quality of your vendors. You might want to make robust code the focus of your policy change long term, and the results you want will follow.

To this end, you could use the key-switches to your advantage. Put all the controllers in run mode, take all the keys and give them to a central librarian. The librarian could enforce your policy regardless of whether the programmer feels it is justified or not. In any case, it should be unnecessary since they are only *looking* at the code, not rewriting it as above, right? :-D

..and yeah, give us an update when you get a chance!

 

[FrancisL]

First raise a change note, with a reference number, then change the documentation, then do the code changes!

Provide the documentation online so that operators can raise the change requests based on the documentation.

Then yes, use "difference detection" -you can do it in various ways, eg a good source code control system (dont know subversion, must look,) or creating a before and after text export and using compare software such as examdiff.

Wot, no documentation?