Confessions 24

[D Scullion]

Many interesting links shared detailing other peoples screw ups. Anybody on here willing to admit to their own?

Not trying to entrap anybody, don't admit to anything you could still get prosecuted for. Just want to hear a few funny stories.

I'll start: My first job in a small engineering firm with very little senior experience, almost every day was a near miss:

- Prototype 24 ton dump trailer, my first folly with hydraulics, on placement from uni. I designed the small hydraulic rams that open and closed a gull wing style back door to operate on the same circuit as the very large rams that lift the 24 ton laden body, without any flow controls. On first operation the back door closed so hard and fast it damaged the solid steel door.

- Ordered a laser cut sheet steel kit for a first production batch of a new product. Didn't realise AutoCAD was set so that all files were scaled 10:1. Received a delivery of miniatures.

- Made a tolerance error on the pivot of a large arm that moved a prototype piece of machinery into operating position. It seized up so tight only the hydraulic ram on the arm could turn it, couldn't even get grease into it. Prototype was on a tight deadline for a demonstration in front of a large group of customers which we had to go ahead with. Started to move the arm and the screeching from the pivot was mortifying. needless to say we waited until everyone had left after the demo before folding it back in again.

 

[waross]

In a standby generator room, typically the discharge from the radiator is through gravity operated dampers that open under the pressure of the discharge air.
The fresh air intake dampers are held closed by a damper operator. Normally open, held closed by power.
The damper motor is on grid power so that when the power fails the damper starts to open.
I heard of a first time startup where the damper motor was configured, normally closed, power to open.
Part of the engineering team was in the generator room to witness the first start.
The generator started and everything loose, dust, paper, whatever started swirling through the air and out the discharge dampers.
The group tried to leave but the low air pressure was holding the door shut.
The generator came up to speed, stabilized and the transfer switch transferred and picked up the load. Then the fresh air damper motor was powered and started to open and let some air into the room.

Bill
--------------------
"Why not the best?"
Jimmy Carter

 

[ScottyUK]

During my first year on a power plant an urgent call came through to our department's morning meeting where jobs for the day were assigned. A circuit breaker hadn't closed when commanded by the plant DCS, could we provide urgent assistance? The switch house in question was right over at the far side of site, a good fifteen minutes brisk walk away. Two of us went down there to investigate, only to find the problem VCB standing in the middle of the switch house floor. Micky's radio transmission back to the control room: "Hey marra, y'know this breaker that you can't close? Have any of you considered racking it into the f***ing switchboard before trying to close it...?" Embarrassed silence.



A few years later I responded to a callout that one of the steam turbine generators had run up, closed onto the grid and immediately over-sped. As anyone with a power generation background will know, this is simply impossible - one machine can't speed up the grid. What the hell have they done...?

A less-cheerful-than-normal Scotty arrives at site in the early hours of the morning and starts checking through the event history and trends for the generator. Everything looks normal - speed, voltage, governor positions - right up to the moment when the generator circuit breaker closes, at which point the governors open slightly to boost the machine from sync idle to minimum stable load... except there's no load indicated and the machine takes off like a scalded cat until it trips at 3300-odd rpm. What the hell have they done...???

The plant control system was pretty well integrated and the DCS did a lot of thinking on behalf of the plant operators, preventing silly errors. Among the things which weren't integrated at this time were the substation supervisory computers, because these belonged to the transmission system operator and not the power plant. Our generators each synchronised across a 275kV breaker in the transmission substation, and in addition to this circuit breaker were three additional HV switches - two busbar disconnectors and one line disconnector forming each switchgear 'bay'. The substation event history recorded that the auto-synchroniser had commanded the circuit breaker to close, and upon seeing the breaker closed status the generator controller had bumped the governors open, but the line disconnector was actually open because the shift guys had forgotten to close it. Based on the information it could see, the machine thought it was on the bars but it was actually running light.

There was a fairly colourful exchange of words with the individual who forgot to close the disconnector because he insisted that he had done so, and that the event recorder must have 'forgotten' to record this happening even though it had recorded the same disconnector opening a few days previously. I'd have had a lot more respect for him if he'd just admitted he made a mistake.

 

[byrdj]

"…the moment when the generator circuit breaker closes, at which point the governors open slightly to boost the machine from sync idle to minimum stable load... except there's no load indicated and the machine takes off like a scalded cat until it trips at 3300-odd rpm. What the hell have they done...???

The plant control system was pretty well integrated and the DCS …"

This confirms my concerns I was having during my last days in the field. I never did a digital turbine control system, with my only acquaintance solving specific problems relate to their implementation on older system.

Trying to “learn” the DCS from conversation with their expert never gave me the feeling that speed control overrode all other functions. That was The Golden Rule

If the incident had been one of the analog controls I worked, the synch demand to load to say 10% would have only allowed an increase above speed of (0.1 demand X 0.05 regulation X 3000 rated =) 15 rpms

 

[ScottyUK]

This was a Westinghouse machine with a DEH control system based on the WDPF II platform. Overall it was a very good system, but there were a few flaws. The generator CB 'closed' status was the signal which took the machine from speed control to load control. Once in load control the speed controller was effectively switched out other than the supervisory overspeed protection functions which were always active.

 

[IRstuff]

Speaking of smoke, two seminal events in freshman year solidified my EE subdiscipline. I built a Plastic Tiger stereo amp, which had a hand-wound damping resistor on the output. The first time I powered it up, the resistors ignited rather spectacularly.

The 2nd was the debugging of a color organ; I hooked up the ground on my oscilloscope, and a 741 op amp exploded from its plastic package, not unlike the creature from Alien, but that was to be for another 6 years.

So, I learned that anything over 5 V was clearly dangerous, and that designing logic ICs might be a pretty safe career choice.

TTFN (ta ta for now)
I can do absolutely anything. I'm an expert!

https://www.youtube.com/watch?v=BKorP55Aqvg

faq731-376 forum1529 Entire Forum list

http://www.eng-tips.com/forumlist.cfm

 

[ScottyUK]

Another event which occurred in the transmission substation I mentioned above was my inauguration into the '900 Club', a far-from-elite bunch of people who had managed to trip an entire 940MW CCGT generating module off the system at one go. We were working on a teleprotection system which essentially multiplexed a pilot wire relay, a duplex 8-way inter-tripping scheme, and a number of voice-frequency signalling channels all down a single optical fibre pair. The voice frequency channels on this thing were the subject of much voodoo hoojoo when I joined, and it basically didn't work because no one really understood how to set it up and after years of tinkering very little of it worked as designed. I'd spent a lot of time replacing damaged components and then calibrating the boards and programming them, and things were going rather well.

One of the features of this system was that the duplex inter-tripping scheme was pretty quick by the standards of the day. It took in signals through optically-coupled inputs and provided a solid-state relay contact at the remote end. One of the factors which wasn't properly considered in the design was that in a large substation there's a fair bit of capacitance between circuit conductors, and between conductors and earth. Capacitors store electric charge, and under the steady-state conditions on a system powered by DC the charge is stored without anyone really being aware of it. If something happens to disturb the steady-state conditions - say an engineer inadvertently earths one pole of the substation tripping & control battery - then current flows into and out of those capacitances until some new equilibrium state is reached. That's fine in principle, unless some of that current happens to find its way through the high speed inter-tripping inputs...

As my test lead trailed across the cabinet floor there was a colossal 'bang' as five huge 275kV circuit breakers opened simultaneously in the tin shed substation. "What the f--- was that...? Are we alive...? We're gonna get bollocked for this..." By this point the roar of steam from the plant had become deafening as 300-odd MW worth of 100 Bar steam was released to atmosphere instead of going into the turbine. Four gas turbines were on their way down too.
And then the phone started ringing...

 

[itsmoked]

Sounds like great fun Scotty!



Keith Cress
kcress -

http://www.flaminsystems.com

 

[byrdj]

I’ll confess…I took credit for pure luck!

Former employer coerced me to visit a plant where their labor group had just finished a turbine outage. The unit was a make I had never even heard off. The problem was the unit would randomly have 1 MW excursions. During the initial conference call I impressed the customer my saying “who worries about 1MW” to which the reply the unit was 4MW rated.

Flew that night to the coldest part of the US I had ever been. On sight I found the drawings were in German and even more confusing was they used symbols I had never seen. The unit was a screamer, the size of an oil drum spinning at scary (to me) rpms. I spent the morning looking at the control cabinet and drawings with no idea what I was looking for.

I took a break and walked around the unit thinking how I was going to tell the owner and my company I was lost, afraid to continue and was going home.

I then heard a change in pitch and looking at the unit, I was in perfect alignment to notice a flash of daylight between a control valve stem and its actuator. The Operator paged me an event was occurring and I quickly went to the cabinet, spending time using a DVM pretending. I then reported that the electronics were responding properly and not the source of the problem. The reason was with the turbine itself, high probability something loose on the #3 CV actuator.

Shut the unit down, “found” the broken stem. I was a hero to the customer, but my employer was not so happy I identified a mistake they made. I left site while a labor crew was returning for repairs.

 

[ScottyUK]

I almost forgot about the day we inadvertently converted a DC pump starter into a (very short-lived) 120kW incandescent lamp.

Each of our turbines was provided with three oil lube oil pumps: the main pump, driven from the turbine shaft; the AC-powered auxiliary pump, powered from the unit substation and used for start-up and shutdown when the shaft-driven pump wasn't at full speed; and the emergency pump, powered from the turbine auxiliary battery. The latter pump was last layer of protection for the rather expensive turbine-generator in the event of a Very Bad Day.

The emergency pump was driven by a fair-sized DC motor, and this motor had a starter of a type which dates back to the beginning of the 20th Century - old-fashioned resistors and contactors, no fancy electronics. The main purpose of the starter was to start the pump while preventing the motor from drawing a destructively high current which would both cause a severe sag in the voltage of the massive battery, and ruin the motor's commutator and brushgear. Ever since the plant was built until I arrived about seven years later, it had been 'just one of those things' that when the emergency pump was tested sometimes the generator tripped because of the disturbance to the control system which shared the battery with the pump. Figuring out what was wrong with these pumps and why they had such an aggressive starting behaviour was one of my first tasks. "Buy tools, buy test gear, buy whatever you need: just get this sorted out".

It became apparent that each pump behaved slightly differently, and also that they were not identically wired. "Eureka!", thinks I, "That's what's wrong with them!" So we set about blueprinting these starters, wire by wire against the schematic and fixing all the anomalies. "Once we sort out the wiring, order will be restored to the world."

Wrong.

We were recommissioning the first of the pump starters and were fairly confident that we'd found the most significant defect: the two series resistances, of differing values, were transposed. We simulated a loss of oil pressure and the pump started. Well, it kind of started. Remember the end of "Raiders of the Lost Ark"? Exactly like that: white light, brighter than the sun, eyeballs melting.

We were transfixed, hypnotised: "F-u-r-r-r-r-r-r-r-r-r-k-k-k-k..... m-e-e-e-e-e-e....."

"Switch it off! Switch it off! SWITCH IT OFF!!!" barked the the gaffer.
I heard the voice of common sense in my head too: "There's a nuclear reaction raging inches above that switch - run!!" and we dived for the safety of the switch house floor.

There was a report like a rifle shot, and the light went out. Silence. And smoke. Lots of smoke.

The underlying cause of the problem, when I eventually found it, was with the field connection of the motor. This motor was a compound design, with a shunt main field assisted by a field in series with the armature. The starting resistors were in connected in the armature circuit, while the shunt field was connected across the full supply voltage, or at least that's where it was supposed to be connected. The schematic actually showed the field connected to the armature side of the starter resistors, not to the supply. The result of this little draughting error was that the motor failed to accelerate properly because the field was only receiving about 15% of its rated voltage even though the motor was drawing about 600 or 700 amps from the supply, followed by a peak of several thousand amps for a fraction of a second as it started across-the-bars when the last resistor dropped out.

The volt-drop caused by the huge transient current was the cause of the machine trips which happened from time to time. It's a tribute to Fuji's motor works in Japan that the commutator wasn't totally ruined by the abuse it had suffered. We wired the field back to where it should have been connected, borrowed a resistor grid from another machine, and it worked beautifully from then on. The guy from Cressall Resistors was slightly bemused when I told him his 120kW lamps didn't last very long, and we needed to buy some more.

 

[Stephen Nuchia]

I still chuckle about this every time it crosses my mind ... a tire plant that has since been re-tired (ha!) had been built during a labor dispute. I don't know all the history but there were some dicey things in there. One Christmas I had to open every load box on all the miles of overhead LV distribution bus, hundreds and hundreds of them, to update the drawings for an arc flash study. Drove around the plant in a manlift for time and a half for a week. In one dark corner I popped a cover open and written inside in sharpie was the old paper matchbook slogan: Close Cover Before Striking. Had to take a break to avoid peeing my nomex.

In an even darker corner of the same plant I found a box that still had a lock on it belonging to a guy only the old-timers remembered. He hadn't worked there for a decade or more, but I was gone before the process of getting that lock removed had run its course. At least they took LOTO seriously.

 

A mixup in metric fuel, we had the Gimli Glider

Gimli Glider

"Everyone is entitled to their own opinions, but they are not entitled to their own facts."

 

[LittleInch]

Scotty Uk = That's a couple of brilliant war stories. Didn't understand the technicalities of most of it, but great story telling.

the only thing I managed to do close to that was wiring a 3P+N generator into the power input of a cabin we were using on site. It looked like a 3 phase supply (Neutral plus L1,L2,L3), but I just removed the links that someone had put in to connect all three because it didn't make sense. Or so I thought.

No wiring diagrams available and I had was a small 3 Phase generator.

Removed the links, connected it up, fired up the generator and went to try out the lights, power and most importantly the brand new wall mounted AC unit. Hmmmm. Some switches managed to work in both directions with varying levels of light, the AC unit kind of started then the control unit went bang and a small stream of smoke emitted from it and a couple of phone chargers got very hot. Once or twice turning the lights on made the generator stall...

In the end a set of sparkys descended on it and re-wired the whole thing so they could have a cool place to recover, once we'd replaced the AC control unit...

Remember - More details = better answers
Also: If you get a response it's polite to respond to it.

 

[waross]

Many years ago, we had a problem with the conveyor to the behive burner at a large integrated sawmill.
The volume of waste was tripping out the drive motor on the last section of conveyor which dropped the waste into the top of the burner.
The motor in question was a 20 HP "U" frame motor. The O/L heaters were already oversized.
I located a very old 25 HP motor and lined up the electrician and the mill wrights.
When the mill went down for the next break everyone pitched in and had the motor switched out in 10 or 15 minutes.
No more problems, the larger motor took the load and no more trips.
Then about a week later a waste 2"x 6" cut-off fell down and jammed the tail spool.
The "U" frame motors had more locked rotor torque than the "T" frames. The old motor was a pre-"U" frame with even more locked rotor torque.
Instead of tripping on overload, the motor kept pulling and ripped the head spool off the supports and damaged the supports.
With the waste conveyor out of service the waste backed up and shut the mill down.
Fortunately this happened on a Friday afternoon.
We only lost about 10 or 11 hours.
It took until about mid-day Sunday to put the fire in the burner out before repairs could start.
Repairs were completed in time for start-up Monday morning.
One minute down time equaled about two days pay for a shift worker.
Our down time was about 5 months pay for an hourly worker.
If this had happened early in the week our down time could have been reckoned in years of the hourly rate.
A few weeks later, over a beer, my supervisor remarked;
"I kind of wish that you had never changed that motor."
Nothing else was ever said.
The larger motor was left in service.
As part of the repairs, the top end of the conveyor was rebuilt to withstand the torque of the 25 HP motor.


Bill
--------------------
"Why not the best?"
Jimmy Carter

 

[RVAmeche]

We'd just gotten into startup and commissioning of a new chilled water system with decent sized headers (10 or 12"). Kicking on the pumps and start actuating valves to the chillers and suddenly it seems like the room is about to shake itself apart and the pipes jump pretty significantly.

Turns out the solenoid valves on the actuated butterfly valves didn't come with the specified exhaust mufflers to slow down their actuation. Whoops. Luckily none of the supports or equipment was damaged by the water hammer but some shorts may have been replaced.

 

[MatthewDB]

Our generators each synchronised across a 275kV breaker in the transmission substation, and in addition to this circuit breaker were three additional HV switches - two busbar disconnectors and one line disconnector forming each switchgear 'bay'. The substation event history recorded that the auto-synchroniser had commanded the circuit breaker to close, and upon seeing the breaker closed status the generator controller had bumped the governors open, but the line disconnector was actually open because the shift guys had forgotten to close it. Based on the information it could see, the machine thought it was on the bars but it was actually running light.

There was a fairly colourful exchange of words with the individual who forgot to close the disconnector because he insisted that he had done so, and that the event recorder must have 'forgotten' to record this happening even though it had recorded the same disconnector opening a few days previously. I'd have had a lot more respect for him if he'd just admitted he made a mistake.

My first boss after University was previously a startup engineer for Bechtel. He shared the story where they had a breaker and a half substation for the power plant, so the generator had two breakers connecting to the grid. During warmup, they would open a disconnect and cycle the breaker to make sure the trip circuit worked properly. Because the generator was still warming up, the sync check relay had to be bypassed to allow the breaker to close.

In this case, they had elected to do each breaker one at a time. The problem was that the guy sent to the switch yard had opened the wrong disconnect. When they closed a breaker from the control room, it closed the unexcited and sub-synchronous generator onto the line. The breaker rapidly tripped, but not before doing extensive damage to the brand new generator.

 

[waross]

My last trip to Central America I rode with a one armed, one legged taxi driver.
I had known him previously as the best and smartest line man who had worked for me.
His partner had opened the wrong disconnect and he had completed a 7.6 kV circuit with his body.
His ex-partner gave him a lot of help getting a new taxi.


Bill
--------------------
"Why not the best?"
Jimmy Carter