Synchronize Generator with UPS

[tg103]

I had a situation occur this weekend at our data facility. We lost utility power to the building for about 5 hours. The generator started ok and picked up the load.

Unfortunately, the UPS was not able to carry the servers during this 10-12 second ATS transfer. It was discovered both UPS battery cabinet breakers had tripped (bad batteries) as well as blown bypass fuses. The servers crashed and we lost everything.

I immediately bypassed the UPS with the MBP and got the PDU's powered. The IT group was called in and went through the process of starting this large data center back up as soon as possible. Meanwhile the UPS tech arrived, identified and replaced the failed components, and about 3 hours later was ready to get the module back on line. By the time he was ready all the servers were back online and utility power was available but left off until the UPS could handle the transfer back.

Now came the problem, we could not get the UPS to sync with the generator to allow a transfer. The frequency of the generator was up and down only fractions but not long enough to initiate a transfer. The generator is a two year old 480/277V 2MW CAT Diesel which was loaded about 50%.

It was decided to take all mechanical loads off of the generator and it helped quite a bit but still not enough time as the UPS was still leading the bypass by 7 degrees. We also had to take down the lighting in an attempt for the generator to freewheel to stabilize the frequency. A painful process.

Finally we had an ok to transfer the UPS and all went well. We restored utility power and the UPS carried the ATS transfer ok.

My question is, is there anything that can be installed to either the generator or service to prevent us from unloading the generator to this extent if this ever occurs again?

Thanks much for any help.

 

[waross]

I am assuming an open transition. That is, the generator disconnects from the load. Then the mains device closes and connects the load to the utility mains. The job of the UPSs is to carry the critical loads until the power is restored.
You said that the UPSs had been repaired. If they won't carry the load for the time it takes to do an open transfer you either need better UPSs or a better repair man.
The procedure should be;
Power fails;
UPSs take up the critical loads,
Generator starts and supplies the main panel.
The UPSs sync with the generator and go on bypass.
Power is restored;
The generator breaker opens.
The UPSs take up the load.
The mains breaker closes.
The UPSs sync with the mains power and go on bypass.

I suspect that the UPSs are inappropriate for the service.
I further suspect that the UPSs took up the load, but were unable to sync with the generator so as to be able to transfer the load to the generator. Eventually the batteries were drained. (I also suspect that the batteries were drained and the bypass fuses blown as a result of a failure to transfer to the generator.)

Suggestions;
Even though the UPSs don't want to sync with the generator, they should have no technical problems maintaining the battery charge with a less than perfect frequency. I mean it's a battery charger and battery chargers are very tolerant of small frequency excursions.
If the UPSs can't tolerate small frequency excursions on the generator, don't try to put the servers on the generator. The UPSs will be going back online everytime a motor starts.
Inhibit the UPSs from transfering to generator power. Let the generator power charge the batteries and leave the UPS inverters running to supply the servers until utility power is restored.
I am surprised that dumping load stabilized the frequency. I find the frequency of sets from about 30 KW up to over 1MW is more stable with more load.
Although autogenous governor mode corrects the frequency to 60 HZ. there must be an error for the governor to respond to. That means that every time the load increases or decreases, there will be a frequency excursion that the governor will correct back to 60 Hz.
It sounds as if you have an electronic autogenous governor.
I would either have the UPS modified so that the batteries can be charged from available power regardless of frequency, or change to a set of UPSs that can handle the duty.
As for adding equipment to the generator or the service, I believe that the problem is with the UPSs they are too delicate to work and live in the real world.
respectfully

 

[alehman]

Internally the UPS is trying to synchronize the inverter to the bypass input voltage. Some UPS' are sensitive to the rate of change of frequency (slew rate) seen at the bypass input. In this case you won't get an 'OK to Transfer' unless the UPS thinks the generator frequency is sufficiently stable for a sufficient period of time. Small but rapid variations can exceed the ability of the inverter frequency control. The UPS O&M manual should provide those specs.

You mentioned that the generator frequency was 'up and down only fractions'. Do you know by what amount, and over what interval of time?

I don't know why reducing the load on the generator would improve the situation unless the engine speed control was having some type of problem at that particular load level. I have seen governors have problems with hunting as load approaches the limit of the engine.

Also keep in mind the voltages must match within a specified tolerance as well. Generator voltage regulators can have problems if the voltage waveform is excessively distorted. Perhaps the HVAC loads have drives that were causing waveform distortion (seems unlikely).

 

[itsmoked]

I'd have the system modified so I could test the UPS and the generator on a regular basis.

Keith Cress
Flamin Systems, Inc.-

http://www.flaminsystems.com

 

[ScottyUK]

I've seen this where the UPS vendor has put some pretty tight requirements on the bypass rate-of-change-of-frequency. Check to see if this parameter can be opened up to allow greater leeway on the behaviour of the generator. Sme UPS designs accept an 'running on generator' contact which commands the UPS to relax some of the power quality tolerances and sometimes reduce battery charging current.

How did the UPS get in to such a poor state without anyone noticing? An auxiliary switch on the battery breaker, some basic monitoring and an occasional inspection isn't much to ask.

Does you generator have utility paralleling capability? If it does then perhaps the easiest way to stabilise frequency is to synch to the utility network until you can sort out your transfer. Most big sets have the capability built in, even if it is not enabled. Of course the utility has to agree with you paralleling to their system which is usually the big hurdle.


----------------------------------

Sometimes I only open my mouth to swap feet...

 

[tg103]

Thanks for the quick responses.

waross - yes, the ATS transfer is an open transition. The entire 2500A switchboard to the facility is backed up from the generator. We have performed bi-annual load tests to this generator by simulating loss of utility power 5 times since commissioning the new genset and never had a problem with the batteries carrying the load for the 10 second open transition.

alehman & scottyUK - your senarios are correct. During these load tests we never had an issue, the batteries carried the load and once gen was on the building the inverter was back online and the UPS ran normally whether or not it was gen or utility input. The problem we had was starting the UPS module from offline with the available generator power.

I questioned this tight tolerance as well. The frequency varied between 59.8 and 60.2Hz about every 5 secs. Each time the UPS display would just toggle between "ok to transfer" and "not ready to transfer". We did not want to take the chance of transferring the rack load back to the UPS until it was steady. Unloading the generator did the trick and I can't explain why.

From what I am told the UPS and batteries are tested quarterly with the last visit in December. Shame on the IT group that handles the maintenance because the bad batteries were flagged and scheduled to be replaced next month. It goes to show how critical it is to replace ASAP I guess.

I totally agree that the UPS must have a setting to widen the variation of this slew rate from the generator. This is where I will start.

Thank you so much for the replies and education.

 

[bjenks]

Can you give the manufacture of the UPS? If APC Silcon then I can help explain the problem. If Liebert, I might be able to also. Please keep in mind that although there is a sync of the frequency for the inverter it will only track +/-3% and will then disable the bypass, break free and provide it own sync +/-15%. It might not go off inverter until it sense a stable frequency. Depending on the maunfacture and technology of UPS could change what I just mentioned. I also wonder if there is any type of passive harmonic filter on the UPS. If an active filter, it is designed to disable at medium to low loads. I wonder if the filter is not working correctly. Input voltage distortion can also cause inverter not to sync. The distortion could be from the UPS input filters, generator gov, or inductive/capacitive harmonic loads. You might need to get a Power Quality engineer out there to monitor all of this. Unless you know why this happened you can't buy anything to solve it.

 

[tg103]

The UPS is a 5 year old Liebert 225KVA 480-480, 2 battery cabinets with (30) UPS12-475 batteries in each. The system is currently 50% loaded.

We have never had a problem with the inverter going to bypass while on generator power since commissioning this new generator a little of two years now. Previously a CAT 1500KW natural gas unit (now the redundant backup) also had no ill effects to the UPS when utility power had failed.

I figure we have a unique situation whereas we never had to start the module on generator power before.

We have bi-annual building load tests and run the building on generator for 7 hours each time.

Quarterly power quality spot checks at the PCC result in low harmonic distortions well within IEEE 519 guidelines, although never measured while on generator power.

Do you think it is worth attaching our power quality analyzer for a few weeks to the main service to get an overall picture?

Thanks for your help.

 

[itsmoked]

I'd run those generators 2 times a month minimum.

Keith Cress
Flamin Systems, Inc.-

http://www.flaminsystems.com

 

[alehman]

59.8 to 60.2 is not unreasonable performance for this type of genset. Perhaps there was something with the HVAC load that was varying or cycling and causing frequency variation.

Your experience with the VRLA batteries is very familiar. They have a tendency to fail suddenly without showing much sign of degredation (and usually right after the replacement warranty period expires). Usually they go to an open circuit. What kind of tests did they do on the batteries? I would recommend at least quarterly internal impedance tests on each cell. With VRLA batteries, you are better to a have several strings in parallel, so if you have an open cell you only loose one string of capacity.

 

[bjenks]

If this is a Liebert with an SCR or 6 pulse front end, passive power factor correction module that is 50% loaded I would bet you had some type of resonating harmonic issue. You need to rule out this out first. There is a setting on the UPS that would automatically disable the PF module at low loads as it has shown many times to cause a harmonic issue that makes it so the UPS will not syn with the generator. However, 2MW generator is a pretty stiff source. I believe this was a distorted waveform that cause the UPS to sync to its internal clock and not the bypass. Once you took the loads off the waveform loss it distortion until it was able to sync the bypass IMHO...

 

[alehman]

I think the filter disconnect function was added some time after about 2000. A Liebert technician showed me one time how to field install a (unapproved) filter switch on earlier 600 series units.

I mentioned the voltage concern also, because I've seen these filters cause havoc with genset voltage regulators. When lightly loaded and the filter on, the input power factor can become leading. If there are insufficient other lagging power factor loads to absorb the VARs, the genset voltage regulator can become confused as it reduces the excitation current to zero, attempting to hold the voltage. This can also lead to false activation of loss of excitation protection.

 

[waross]

Your frequency variation troubles me. Do you have a load that cycles every 5 seconds? How about time proportional electric heat control. I do agree that your frequency range is good but the 5 second cycling is bad. You may have to tweak the stability adjustment of the governor. I would phone the Cat service manager and discuss the hunting frequency with him. It may be that a quarter turn or less of an adjustment screw will stabilize the frequency.
I was on a mine construction project years ago that started on diesel power until the transmission line construction was finished. One of the generators was throwing a crank about every 6 weeks. It turned out that a time proportional controller for the electric heating in the office complex was causing the crankshaft problems. The problem was solved by jury rigging a conventional thermostat to control the heating controller until utility power was available.
The point?
A hunting governor means extra stress on the crank and bearings.
A fast cycling load will probably cause more stress on the machine.
The recommendations?
Talk to Cat about stability adjustments on the governor.
Check your loads for a fast cycling load such as time proportional electric heating. If the frequency hunting is load induced, you may be able to stretch the cycle time from 5 seconds to several minutes during the time that you are on the genset.
respectfully

 

[tg103]

Waross good point. Yes the electric heating is controlled by discrete relays and after shutting them down as well as many sewer ejection pump systems, it greatly reduced the generator from hunting and allowed us to eventually synchronize the UPS.

I will investigate that option you mentioned and perhaps adjust the timing between cycles and also look into adjusting the floats on the holding tanks.

The IT group has scheduled complete battery replacement for this Sunday and the owners have agreed to simulate the condition we experienced last weekend in a few weeks during a maintenance window.

I do have one last question in regards to itsmoked recommendation. I agree to perform transfer tests more often than twice a year. From what I understand about these high rate VRLA batteries is that these multiple discharges will greatly reduce their life expectancy. Am I safe to assume this?

I'll keep tuned in.

Thanks much for the help.

 

[waross]

It looks like you have things in hand. Let us know how it goes.
By the way, if you have power factor correction, beware of the power factor going leading on light loads when on the generator. This will overexcite the generator and cause high voltage issues.
respectfully

 

[alehman]

Hopefully your batteries are designed for high rate UPS applications. High rate discharge is definitely hard on batteries - whether designed for UPS or not. I generally try to size batteries for at least 15 minutes for that reason, even if I don't really need 15 minutes.

One manufacturer of large UPS systems has taken the approach of sizing batteries for only a few minutes. To counter the effects of high rate, they have a periodic battery replacement program.

 

[rob46]

Please review article "Applying Uninterruptible Power Supplies: System Compatibility Isues" by Thomas M. Blooming, Harold Ketola and Daniel H. EGLIAN.

 

[joan271273]

I had a similar situation , the UPS were 500 kVA and the culprit was our low level of load within the space. When we load bank at specified loads it all went weel. The day we needed our low level (<50%) did not allow us to get the system sysnc

 

[mpparent]

joan271273,

Your issue probably had something to do with an input filter on your UPS. Some filters will drop out under a certain load so that there aren't issues with the UPS and generator "getting along".

Mike

 

[powerwagon75]

From the "for what its worth", I agree with an earlier post and would scrutinize the genset's governor settings. You stated this is a 2MW diesel, running at about 50%. I would not expect to see the .4 Hz swing on that genset; possibly if the stability and/or gain were a little off, and the load that is cycling is pretty much all of the 50% it's carrying. Have someone who is familiar and competent with the genset's governor controller run through the adjustments for you. Your controller probably has more than just the gain/stability adjustments, but these two are very key to load response. You don't want the response so fast, that the fuel response is "jiggly" (rapid small surges), on the other hand you don't want it set up too slow either. Slow will give you a nice, steady engine at constant load, but will provide terrible load change recovery rates (slow recovery time & over/undershoot Hz errors.)
I don't have nearly the UPS experience the previous posters do, but the ones I've been involved with that had symptoms you described were solved by the UPS folks making slew settings wider. Some were voltage regulations issues (almost always with smaller gensets) that involved replacing voltage regulators for better quality and/or adding PMG regulator power supply.

Thanks for letting me ramble and good luck (if not already solved.)