Callide Power Station C4 Generator Failure Update 6

[EnzoAus]

thread815-483510

CS Energy has released its technical report into the 2021 C4 424MW steam powered generator failure of May 2021. At least one other independent report is expected to follow. The report is available at

https://www.csenergy.com.au/news/cs-energy-releases-technical-report-into-2021-unit-c4-incident

and includes a 20min video and a PDF.

There was a cascade of failures, but the triggering event was the collapse of the 220Vdc bus voltage at Unit C4 after the disconnection of two paralleled battery chargers. The well presented video explains that rather than sharing load, two chargers operating in parallel during commissioning, did not share load, but effectively loaded the one with a slightly higher voltage. When the two chargers were disconnected, the DC voltage from the new charger supplying the C4 unit collapsed. Pages 7 to 9 of the report list the sequence of events and include a voltage graph.

The report summary of contributing factors listed (my order):
1. A DC interlocking design that disallowed two independent batteries to be paralleled, thereby requiring chargers to be paralleled for DC redundancy
2. Battery charger/s that failed to share load when in parallel in a manner that allowed them to continue operation when separated
3. Unintended operation of "arc flap" (sic - arc flash?) protection that tripped AC to the generator auxiliary systems as a result of the DC failure
4. Failure of the DC automatic changeover switch which had been damaged 4 months earlier and presumably not been repaired

The combined AC and DC failure led to loss of lubrication oil to turbine and generator bearings and sealing oil which normally prevents hydrogen coolant from escaping. The hydrogen is believed to have combusted, the bearings failed and the generator ended up motoring at 300MW/1400Mvar until it flew apart. The Callide operators had dark SCADA system screens and were unable to determine exactly what was happening in real time.

I can't help wondering:
1. whether the 220Vdc battery chargers were designed unable to share load in parallel (which seems unusual) or whether one had a fault and
2. whether Powerlink, the transmission network operator did not see the power reversal and extraordinary reactive power flows which could have alerted them to the critical event unfolding.

 

[EnzoAus]

The draft Brady Heywood reports into Callide C4 incident have been released and are available from CS Energy's website.

https://www.csenergy.com.au/news/release-of-draft-brady-heywood-reports-into-callide-c4-incident

A quick recap of the technical causes for those needing a refresher:
[ol 1]
[li]Switching procedure relied on a replacement battery charger to quickly pickup full load from a DC supply that was being disconnected.[/li]
[li]The battery charger had not been specified to pickup load quickly and failed to do so.[/li]
[li]A changeover switch designed to provide redundancy had known blown fuses and was left inoperative for the switching procedure.[/li]
[li]Failure of the DC led to a trip of AC systems leaving the unit motoring until it blew apart.[/li]
[/ol]

I've only had a quick look at the Exec summary. The organisation cops significant criticism including this in the summary on p11. "In this context, the failure to understand and assess risk, and to not effectively apply sound management of change processes in relation to the engineering factors that led to the catastrophic failure, suggests these were not isolated incidents, but rather a symptom of an organisation’s failure to value and implement effective process safety practices."

Another critical quote regarding CS Energy's method of dealing with previous incidents. "The assurance system only monitored completion of those agreed actions, and did not consider whether those actions had improved the effectiveness of the system."

Many of us have seen similar box ticking operations that fail to deal with the underlying issues. I wonder how many real engineers within the organization have been excluded from playing a significant role in the management decisions. We can only imagine how many other generation, transmission and distribution entities are running with the same types of risks happily masked by glossy annual reports.

 

[Richard Baum]

The C4 220VDC was isolated with the unit in service, a written high voltage isolation was not used, the insurance has not made a payment to CSEnergy,
The RTS insurance will be self insurance by Qld Gov, their is a problem with staff turnover, there is a problem with operator training, maybe the CS Energy mess should be given to Snowy Hydro as surely they would not turn off the 220VDC. They provided a picture of the C4 220V battery, is that small battery able to supply all the critical services for the C4 unit for one hour with battery charger not available. Does the employment of a extra 40 Engineers help, remember the C4 unit operator did not know that that the 220VDC had been isolated

 

[Richard Baum]

Beleaguered state-owned CS Energy spent AU$38m of taxpayers' money hiring lawyers over the past two years, including to fight the release of the landmark report into the company's woes. The investigative report into the 2021 Callide C explosion compiled by forensic engineer Dr Sean Brady, cost taxpayers a further AU$10.4m.

 

[MintJulep]

The investigative report into the 2021 Callide C explosion compiled by forensic engineer Dr Sean Brady, cost taxpayers AU$10.4m

Seems a bargain for a comprehensive report by a leading forensic engineering firm about a complex chain of failures.

Now someone with authority just needs to read it.

 

[human909]

Now someone with authority just needs to read it.

You forgot the part about understanding and caring to take action to improve outcomes!

That is often the thing about institutional bureaucracies. People are just looking after themselves and working within the bureaucracy rather than having a real desire and drive to improve. They want to be SEEN to be getting the job done first and foremost, and only secondary is actually doing the job.

I got a good taster of that the other week. I went into a meeting where the government client had "a few questions" for the structural engineer. It was an ambush. Half a dozen project engineers and and project managers were present to put me on the spot with aggressive questioning. They were fully prepared with plenty of bluster. However their problem was they had little idea of what they actually wanted and didn't have the knowledge to ask suitable questions.

A waste of time for all involved. But I presume working in some government bureaucracies is about filling your day with meetings and going home early.

 

[MintJulep]

You forgot the part about understanding and caring to take action to improve outcomes!

Isn't that what "authority" means should mean?

authority
noun
1. power to influence or command thought, opinion, or behavior

 

[Richard Baum]

Nothing complex about switching off the critical 220VDC with C4 in service, just ignorance and good reason for no au$300M insurance payment.
CS Energy’s process safety management systems were not where they needed to be and the critical 220 VDC supply to unit C4 was turned off by maintenance staff with unit in service without the operators that where in charge of the power plant operation knowing. Then things were very complex with no income from C4 for over three years. Management even gave them selves a bonus after the coverup.

 

[EnzoAus]

For completeness, the final Brady Heywood Report can be found at

https://www.csenergy.com.au/news/release-of-brady-heywood-report-and-c3-cooling-tower-report.

It's a 28MB download BUT the appendices have not been included!

It may be interesting for some to note the issue with the particular battery charger used in the C4 unit. When paralleled with another charger of higher voltage, rather than share load, the C4 charger produced no power and the internal voltage decayed to zero (Ref p113, Sn 9.6.1). This explains why when the charger was decoupled from the paralleled DC bus, it was not ready to support the load and the DC voltage collapsed.

 

[Richard Baum]

“The Callide Unit C4 incident was the result of the simultaneous failure of key electrical equipment and system back-ups in a complex series of events that could not have been anticipated, with some of the contributing factors being traced back to the original design of the power station,” Mr Busine said.

What is the total cost $50M on coverup, maybe $500M on rebuild because the Insurance has refused a payment all because CS Energy allowed maintenance staff to switch off the critical 220V battery with the C4 Turbine in service.

The C4 operator in the central control room did not know that the 220 battery had been isolated.

 

[GregLocock]

Is there an FMEA? If not did various investors do due diligence when investing?

Cheers

Greg Locock


New here? Try reading these, they might help FAQ731-376

http://eng-tips.com/market.cfm?

 

[Richard Baum]

The owner is the government of Queensland in Australia
The Callide Unit C4 incident was the result of the simultaneous failure of key electrical equipment and system back-ups in a complex series of events that could not have been anticipated, with some of the contributing factors being traced back to the original design of the power station,” Mr Busine said.

This is classic technical writer cover-up “gobbledygook”!

$50M is the cost of the legal advice and inquiry which was meant to be secret by claiming legal privilege after maintenance staff isolated the critical 220V battery supply with the C4 Turbine in service so the insurance was voided!
The C4 unit operator “who most likely now has PDSD” in the principal control room did not know that the 220V battery had been isolated, so the C4 Alternator motored to complete destruction!