Callide Power Station C4 Generator Failure Update 6

[EnzoAus]

thread815-483510

CS Energy has released its technical report into the 2021 C4 424MW steam powered generator failure of May 2021. At least one other independent report is expected to follow. The report is available at

https://www.csenergy.com.au/news/cs-energy-releases-technical-report-into-2021-unit-c4-incident

and includes a 20min video and a PDF.

There was a cascade of failures, but the triggering event was the collapse of the 220Vdc bus voltage at Unit C4 after the disconnection of two paralleled battery chargers. The well presented video explains that rather than sharing load, two chargers operating in parallel during commissioning, did not share load, but effectively loaded the one with a slightly higher voltage. When the two chargers were disconnected, the DC voltage from the new charger supplying the C4 unit collapsed. Pages 7 to 9 of the report list the sequence of events and include a voltage graph.

The report summary of contributing factors listed (my order):
1. A DC interlocking design that disallowed two independent batteries to be paralleled, thereby requiring chargers to be paralleled for DC redundancy
2. Battery charger/s that failed to share load when in parallel in a manner that allowed them to continue operation when separated
3. Unintended operation of "arc flap" (sic - arc flash?) protection that tripped AC to the generator auxiliary systems as a result of the DC failure
4. Failure of the DC automatic changeover switch which had been damaged 4 months earlier and presumably not been repaired

The combined AC and DC failure led to loss of lubrication oil to turbine and generator bearings and sealing oil which normally prevents hydrogen coolant from escaping. The hydrogen is believed to have combusted, the bearings failed and the generator ended up motoring at 300MW/1400Mvar until it flew apart. The Callide operators had dark SCADA system screens and were unable to determine exactly what was happening in real time.

I can't help wondering:
1. whether the 220Vdc battery chargers were designed unable to share load in parallel (which seems unusual) or whether one had a fault and
2. whether Powerlink, the transmission network operator did not see the power reversal and extraordinary reactive power flows which could have alerted them to the critical event unfolding.

 

[TugboatEng]

I have noticed a few design deficiencies on DC backup systems on tugboats:

Isolation by rectifier or automatic combining relay alone. In the event of a loss of voltage on one bank, paralleling in this causes the failed bank to become a burden on the other bank. This can drag both of them down. The questionable power source needs to be disconnected and not paralleled with.

Change-over relays that operate on loss of primary voltage to transfer to backup. In this instance, the backup relay can latch on to a dead bank if backup is not available. I prefer to power the control circuit for the relay from the backup source. In this case the relay can't transfer to backup power unless it has backup power is available.

No maintaining contacts. This is especially true in the case of flooded lead acid batteries. FLA batteries recover voltage when load is disconnected. Without a maintaining contact the relay can rapidly transfer back and forth between the two power sources damaging the relay.

Using critical loads to back each other up. On a boat with two engines it's standard practice to back up each engine with the other's power. This means a single fault can take down both engines resulting in a loss of control of the vessel.

 

[FacEngrPE]

The term - arc flap protection occurs in two of the reports and the video, so it must be an Australian term.
I think this may refer to the doors on top of the switch gear which flap open should an arc occur inside the gear. The opening of these doors is what was supposed to be sensed.

I was going to make a comment about operating with a failed DC Transfer switch interlock controller long enough that that became a normal condition, however the report states something different.
Efforts to replace the failed part with a new design were in progress, but took too long as the event occurred before the new design was installed.

CS Energy does acknowledge revising a number of business processes as a result of this event, so perhaps they followed the string far enough to fix some of them.

The ability of battery chargers to share load depends on the same sort of decisions that allow paralleled generators to share or not share load. The report indicates that it was decided to change to a battery charger that is designed to share load better. Likely something like adjusting the droop curve. If the droop is too flat load will never share. If it is too steep voltage regulation will be poor, but load will share.

 

[TugboatEng]

I have found the modern electronic style (switch mode) battery chargers to be incredibly unreliable. Mag-Amp only on my boats. I wonder which type of chargers were in use? I have a preferred brand, they may be the only one.

 

[Alistair_Heaton]

Wonder what the aviation industry uses?

Must admit my knowledge which is limited, is a load of acronyms like BCU (battery control unit)

A220 has to deal with 5 independent power sources, 2 main battery's and another 2 I think emergency ones on other systems plus the lighting backups. 3 TRU's with full automatic fault bus and generator isolation.

Even the Jetstream 41 had it.

 

[KevinNZ]

Video

https://www.youtube.com/watch?v=vbLvjFohK9g

 

[Richard Baum]

The cause of this failure and the following loss of more than A$500m with the insurance not wanting to pay was caused by management not providing enough operator training in Operation,Safety and Plant Isolation Proceedure.
Why was the unit in service?
The switching (that caused the complete loss of critical DC supply) was carried out with C4 in service. Who decided it was safe to do so?
There where known faults within the 220VDC switchboards.
Who prepared this very expensive switching? (Two operators and their supervisor?) did they have access to knowledge of past failures?
Did the operator doing the switching understand what he was doing? (was he qualified to carry out such a critical switching)did his supervisor know how critical the 220VDC supply was?
Did the Power Plant Operater (PPO) of C4 know that the switching was about to be carried out?
(A switching that could cause a complete loss of control of C4).
Plant failure and loss of operator knowledge seem to happen in cycles of about every 15 years because of staff movement and retirement.
Operating staff should not penalised for not realising how critical the 220VDC supply was, It is a management problem caused by their failure to provide safe operating proceedure and knowledge

https://www.aer.gov.au/system/files...g - Concise Statement - 9 February 2024_0.pdf

Court proceedings are underway against a Queensland coal-fired power plant accused of failing to comply with performance standards.

The Australian Energy Regulator began Federal Court proceedings on Friday against Callide Power Trading (CPT), which trades the output of the Callide C power plant near Biloela in Queensland in the national electricity market.

The regulator alleges CPT breached two national electricity rules following an investigation into a mass power outage on May 25, 2021.

An explosion in the plant's turbine hall resulted in a trip of multiple generators and high-voltage transmission lines across Queensland, leaving 470,000 homes and businesses without power.

The regulator alleges Callide's C4 unit failed to ensure its plant met performance standards and did not design its facilities to comply with those requirements.

"The failure of Callide C4’s protection systems to disconnect the generating unit from the power system resulted in the trip of multiple generators," the regulator said in a statement.

Regulator board member Justin Oliver said compliance with generator performance standards is critical.

"It’s vital that registered participants and generators are aware of their performance standards and comply with them at all times so that the market and consumers aren’t wrongly exposed to the consequences of adverse events," Mr Oliver said.

The Callide C plant can generate up to 1540MW of electricity - about 30 per cent of the state's overnight demand.

Callide B's two units, which are run by state-owned electricity generator CS Energy, were quickly back in operation following the incident.

The repair of units C3 and C4 has been delayed through 2023 and now into 2024, with their partial return to service expected on February 29 and June 30 respectively.

A spokesperson for CPT said they will work co-operatively with the regulator to resolve the matter as soon as possible.

The regulator is seeking pecuniary penalties, declarations, orders for remedying the breach or preventing the recurrence of the breach, and costs.

 

[EnzoAus]

Richard, some very interesting questions, particularly the decision to separate the C4 charger whilst C4 was in service. Given that the batteries could not be paralleled, simply separating the C4 Battery Charger from the Station Battery Charger & Battery would immediately leave C4 without any battery backup should the charger fail (Fig 5 image from CS Energy report). Such a situation would seem to warrant separating the DC chargers whilst the generator was off line to avoid exposure to a single failure mode.

It is interesting to note that the Australian Energy Regulator alleges, in their filing that you provided, that Callide Power did not provide appropriate redundancy for the protection systems as required by legislation.

The other issue not explained is what was different in this switching activity compared to the previous operation to replace the charger on the C3 generator. One assumes that that was performed successfully. Maybe C3 was off line at the time? It is still not clear whether the charger failed or whether there are inadequacies in the type of charger selected for the role.

 

[human909]

The other issue not explained is what was different in this switching activity compared to the previous operation to replace the charger on the C3 generator. One assumes that that was performed successfully. Maybe C3 was off line at the time? It is still not clear whether the charger failed or whether there are inadequacies in the type of charger selected for the role.

The same thought occurred to me.

Purely from knowledge based on that CSenergy video I would conjecture:
1. The C3 just didn't hit the unlike 'sweet spot' of DC voltage that triggered the cascade. Maybe C3 took the load and STATION lowered voltage. Or mabe the change over was 20s shorter and the voltage recovered without anybody noticing the issue.
2. The ACS on C3 was working whereas on C4 it was not. (This is probably unlikely as, in my limited understanding, had the sequence reached this point it wouldn't have been a smooth change over and lessons would have been learnt.)

 

[Richard Baum]

Wednesday 10 April 2024

1/ The insurance will not pay,

2/ The part owner of C4 has a court order for a new enquiry

3/ The Australian Energy Regulator is taking legal action

4/ A industry standard Castell interlock System had not been fitted to the unit breaker to stop the isolation of the 220 VDC control supply when the C4 unit was in service.

5/ The original forensic enquiry by a structure engineer is on going

 

[Richard Baum]

Indeed, the modification of the Callide C4 design, which included the removal of the Castell key interlocks, unfortunately led to a catastrophic failure. The attempt to improve the system’s redundancy and reliability inadvertently resulted in the complete destruction of the steam turbine, incurring significant costs.

The incident underscores the importance of thorough risk assessment and testing when implementing design changes, especially in critical infrastructure like power plants. It’s a stark reminder that while innovation and improvement are necessary, they must be approached with caution to ensure safety and reliability are not compromised.

The lessons learned from the Callide C4 incident will undoubtedly contribute to safer and more reliable design practices in the future. It’s a costly but valuable lesson on the potential risks of modifying established systems.

Large 220 V Lead Acid Batteries as used in power stations to power motors and circuit breakers can not be connected together because of the large fault currents possible, The castell key interlocks ensure that the critical circuit breakers allways have a 220VDC supply when the unit is in service. The unit CB is always locked open before any 220 VDC switching can take place, that means the unit is out of service.

To remove the critical Castell key interlocking is ignorance
(When an isolation, reconfiguration or load transfer has the potential to impact the 220VDC and 48VDC protection systems for Generator, Turbine, XY protection, GCB protection, AVR excitation and 275KV systems a Switching Sheet is required. When the configuration of the switchboard does not allow for redundancy to maintain the bus load during the switching process, no switching will proceed unless the unit is brought offline (desynched). All switching on these systems must have Site GM approval through the Operational Risk Assessment process (ORA).) is Callide’s way of not wanting to re-install the original Castell key into locking system.
So the Site GM has replaced the Castell Key locking system! because they are trying to hide their ignorance

 

[Richard Baum]

CSEnergy C4 switched off the critical 220VDC with C4 in service and wrecked a steam turbine and three years later the coverup is still on. No wonder the insurance has refused to pay.

Forensic engineer Sean Brady has still not completed his long awaited report into the explosion that crippled Queensland's Callide C power station three years ago, he did not contribute to CS Energy's February report on the same incident, and he did not form an early view on how long his investigation would take.
Those are the only three clear answers from almost four hours of Dr Brady's examination in front of the Federal Court on Monday by lawyers for FTI Consulting's John Park, appointed by the court in January as a special purpose administrator to the collapsed company that owned half of the power station.If there was ever any doubt that CS Energy is desperate to prevent the causes of the incident from being made public, Monday's farce in front of Federal Court registrar
Peter Schmidt made it pretty clear.
Mr Park was given a brief by Justice Roger Derrington to examine the causes of the explosion that crippled the power station, and the causes of the collapse of a cooling tower in 2022.
Dr Brady was hired to do the same job in June 2021, but is still to deliver his report three years later.
But he was prevented on Monday from telling the court why it had taken so long; whether CS Energy or its lawyers had instructed him to go slow on delivering his re-port; whether he agreed to CS Energy's own assessment that the failure was the result of a "series of complex events that could not have been anticipated"; or even whether he had spoken to other independent experts about the explosion and, if so, who.
To be clear, the blockage was not of Dr Brady's making, or his own lawyers, but came from the lawyers advising CS Energy on its own legal position.
[highlight #FCE94F]The reason is that Dr Brady was not hired by CS Energy to conduct his investigation, but by lawyers for state-owned CS Energy, Norton Rose Fulbright (NRF).[/highlight][highlight #FCE94F][/highlight]
That, according to arguments advanced in court, makes anything associated with the conduct of Dr Brady's investigation, the report itself - and even the terms of reference he was given - subject to legal professional privilege, and not available to FTI Consulting.
Or to the households and businesses paying higher energy costs as a result of the explosion at Callide C run by taxpayer-owned CS Energy.
And on the same arguments, instead of a clear-eyed questioning of Dr Brady about the slow progress of his investigation, and any interim conclusions he might have reached, Monday's examination instead turned into a farce, in which NRF's legal brief - Patrick O'Shea KC, one of Queensland's top barristers - was dispatched to court to object to almost every question asked.
Including, at one point, questions about whether Dr Brady had written comments attributed to him in a CS Energy press release
Whether the claim of privilege can be sustained will be sorted out by Justice Derrington next week, but in the meantime it's worth reflecting on what CS Energy has said in public.
When CS Energy announced the appointment of Dr Brady in June 2021, then chief executive Andrew Bills said the forensic engineer "has been given the authority to expand its scope based on progressive findings".
On Monday Dr Brady was prevented from answering questions about that topic, as it might go to the brief of instructions he had been given by Norton Rose Fullbright.
In a press release in July 2021
Mr Bills said Dr Brady and his team had been on site at Callide C, and "we are providing them with whatever support they need to undertake a comprehensive investigation".
Dr Brady was allowed to confirm he had visited the stricken power station, but lawyers for NRF objected to him answering anything on the second part of the statement - although Dr Brady was eventually allowed to say he wasn't quite sure what was meant by "support".
In October 2021, in another press release, Mr Bills said CS Energy would "share the learnings and findings from Dr Brady's investigation with the power generation industry".
Dr Brady was allowed to acknowledge he had seen that release at the time and said that, at that point, he had not formed a view about how long his investigation would take.
In CS Energy's 2022 annual report, published in September of that year, Mr Bills said Dr Brady's report was "ongoing and is expected to be completed in the second half of 2022".
In May 2023, on the two-year anniversary of the explosion, acting CS Energy chief executive Andrew Varvari said in a press release that Dr Brady's "external, independent investigation into the Unit C4 incident was entering its final stages".
Dr Brady was quoted in that release saying his investigation was "being conducted by a broad range of subject matter experts" and included independent testing.
Lawyers for NRF objected to questions about who those subject matter experts were, and whether Dr Brady wrote those quotes himself - he was eventually allowed to confirm that the quotes were approved by him, but were a modified version of lines initially supplied by someone else.
In CS Energy's 2023 annual report the company said the Brady report was in its final sta-ges, and in October new CEO Darren Busine reiterated that timing in a public statement.
"As we have stated previously, we are committed to sharing the findings and learnings from the Unit C4 incident with industry to prevent an incident like the C4 event from happening again." And then, as CS Energy released its own version of the causes of the incident in February 2024, Mr Busine said "we are eagerly awaiting the completion of Dr Brady's investigation".
Which leads to the obvious question immediately asked by lawyers for FTI
Why haven't you finished it And that, yet again, went unanswered after an objection from Mr O'Shea, appearing for Norton Rose Fulbright - the legal advisers to CS Energy.
And why would CS Energy want Dr Brady's report sup-pressed, contrary to its own public statements? Go figure.[highlight #FCE94F][highlight #FCE94F][/highlight][/highlight]

 

[EnzoAus]

Richard,
A 3 year delay in producing an independent report seems unacceptable and the use of legal privilege to retain the rights to that investigation raises doubts about any intended transparency. It's disappointing, particularly given that CS Energy is a publicly owned company.

An online search

https://www.herbertsmithfreehills.com/insights/2021-06/legal-professional-privilege-waiver

reveals that legal privilege can be lost if there has been:
[ul]
[li]intentional disclosure,[/li]
[li]unintentional disclosure or[/li]
[li]implied waiver.[/li]
[/ul]

There has been some detailed information (disclosure) revealed by CS Energy to date, particularly within the link referenced in the original post. During the current procedures, assuming the witnesses are under oath, is it naive to think they could simply be questioned whether any of the released documentation is similar and possibly drawn from any part of the legally privileged investigation? Any answer to the affirmative would presumably open up an interesting new line of questioning.

 

[dik]

In most jurisdictions, that is not possible and as an expert, the court can compel him to testify.

-----*****-----
So strange to see the singularity approaching while the entire planet is rapidly turning into a hellscape. -John Coates

-Dik

 

[GregLocock]

The ongoing court case reveals that as usual Labor Lies. The state government was told back in 2020 that the report would not be made public.

The Queensland government was told by the boss of CS Energy “at an early stage” that the state-owned power company would not release a damaging independent report into an explosion at one of its generators, which led to hikes in electricity prices, despite subsequent assurances from Labor that it was going to be made public.

On Thursday, Federal Court judge Roger Derrington threw out an attempt by CS Energy to keep secret a report into the 2021 explosion that crippled the Callide C power station, ruling that the review by forensic engineer Sean Brady should be released to private investors in the plant looking to sue the state government for mismanaging the plant.




Cheers

Greg Locock


New here? Try reading these, they might help FAQ731-376

http://eng-tips.com/market.cfm?

 

[Richard Baum]

CS Energy CEO Darren Busine said: “This was one of the most significant and complex process safety incidents in Australia and it has taken time to work through and understand the contributing factors to the event. and that the forensic engineer was employed by C S Energy's Lawyer. No wonder the insurance is taking action against C S Energy.


“The Callide Unit C4 incident was the result of the simultaneous failure of key electrical equipment and system back-ups in a complex series of events that could not have been anticipated, with some of the contributing factors being traced back to the original design of the power station," Mr Busine said.

The ongoing court case reveals that as usual Labor Lies. The state government was told back in 2020 that the report would not be made public. Greg Locock said

C S Energy's Callide C4 was wrecked in May 2021 so who told the state government? in 2020?

When an isolation, reconfiguration or load transfer has the potential to impact the 220VDC and 48VDC protection systems for Generator, Turbine, XY protection, GCB protection, AVR excitation and 275KV systems a Switching Sheet is required. When the configuration of the switchboard does not allow for redundancy to maintain the bus load during the switching process, no switching will proceed unless the unit is brought offline (desynched). All switching on these systems must have Site GM approval through the Operational Risk Assessment process (ORA).)

What happened with C S Energy's Callide C4 was ignorance,the critical Callide C4 unit 220VDC Busbar was disconected from both the 220VDC batteries that were available with Callide C4 connected to the grid. No Site GM approval through the Operational Risk Assessment process,No written High Voltage Apparatus Isolation Instruction or switching sheet which had been replaced with a computerized black padlock proceedure, a complex series of events that could not have been anticipated, with some of the contributing factors being traced back to the original design of the power station," Mr Busine said. Most power stations use Castell keyed interlocks which would have been included in the original design to ensure that the critical 220Vdc backup batteries are always available when required.

Ignorance is just a lack of knowledge which can be lost by employees being retired, or Engineers changing procedures that have been developed over many years by experienced operators, coloured padlocks and computorized Isolation Instructions, the crazy operator training were every power company and every state does their own thing is a problem.
Where is the data base of past failures?

 

[zeusfaber]

Labor Lies

Greg, I live too far away to understand this. What's "Labor" in this context? A political party, a reference to what I would call the Trade Unions, or something else?

 

[TugboatEng]

Labor is Australia's left leaning political party.

https://en.m.wikipedia.org/wiki/Australian_Labor_Party

 

[zeusfaber]

Ta.

 

[GregLocock]

Sorry typo 2021 not 2020

Cheers

Greg Locock


New here? Try reading these, they might help FAQ731-376

http://eng-tips.com/market.cfm?