Richmond VA Boil Water Advisory

This video has been confirmed by Richmond Times Dispatch to be Richmond Monday morning despite initial statements to the contrary by the city.

Still no word on what initially failed/caused the flooding. The city is basically running on rental pumps right now with pressure slowly being restored after 4 days.

Quick Mr. Trump.
Turn off the big faucet.

Allegedly the pumphouse had batteries and a UPS with a manual backup generator. Once batteries ran out after an hour or so, the generator wasn't running for some reason, and the pumps abruptly shutting down resulted in the clearwell overflowing and flooding the basement?

Yesterday pressure was mostly restored, after 3 days of tens of thousands of people not having water. Still under boil advisory until testing clears, but at least people actually have water again.

There was a technical discussion last night that basically the sequence of events was:

1. Power outage from primary electrical feed occurred during snow storm. Site has two power feeds and normally automatically switches over when one goes down, for whatever reason switch didn't happen this time
2. Electrician went out to manually switch power. Decided generators didn't need to be manually started.
3. Switch took over an hour and the battery UPS system gave out after 45 mins (system was partially inoperable affecting battery life). This caused the SCADA control system to crash.
4. When the SCADA system came back online, it glitched and wouldn't connect to servers.
5. Due to lack of controls, valves to the filters/clearwell remained open and flooded the pump basement beneath the clearwell.
6. Operators didn't know they could/should manually close those valves.
7. Flood into basement exceeded sump pump capacity out
8. Basement flood level rapidly rose, eventually getting into electrical panels and causing much more severe problems.

To me it sounds like the facility generally got complacent because they had redundant power feeds and automatic switch that allegedly had never failed before. The fact that operators literally didn't know about manually closing valves to prevent the basement from flooding points to this.

Why every procedure should have a checklist.

Every backup system MUST be periodically tested under conditions that mimic the problem it is supposed to mitigate.
These tests need to be designed so that when the test fails, there is a recovery path, or the havoc that results will be same as an unplanned failure.

I would advocate one of the planned tests for a critical waterworks should be a hard power down black start.

FacEngrPE said:

Every backup system MUST be periodically tested under conditions that mimic the problem it is supposed to mitigate.

Click to expand...

Even earlier, the design FMECA needs to actually list these failure modes and suggest mitigation methods.

"Ah, that'll never happen!" isn't an actual mitigation.

MintJulep said:

"Ah, that'll never happen!" isn't an actual mitigation.

Click to expand...

Didn't Denney Pate say something like that?
Other notable Denney quote;
"My phone? Just a minute, the wash is almost done."

Not that it matters now, the Virginia code has requirements that if met should have prevented this waterworks process upset.
Pump Room flooded monday Jan 6
Full service restored / boil water advisory canceled Jan 11
for a short utility power outage. Two primary distribution feeders does not eliminate all common mode failures.
just some of them.
Va Department of Health Richmond Water Plant Response

VDH Waterworks Regulations Now in Effect​

12VAC5-590-50. Application of this chapter to waterworks in operation or
planned before June 23, 2021.
A. The owner shall comply with Part II (12VAC5-590-340 et seq.) of this chapter unless a
variance or exemption is issued by the commissioner.
B. Compliance with design criteria set forth in Part III of this chapter is limited to
modifications to existing waterworks and for all construction of new waterworks commenced
after June 23, 2021. Portions of waterworks not being modified are not required to comply
with the design criteria of Part III (12VAC5-590-640 et seq.). Waterworks construction or
modification is deemed to be commenced for purposes of this section upon issuance of the
construction permit.
C. Compliance with the requirements set forth in Part III of this chapter including those for
materials, construction methods, and disinfection is necessary for all repairs to pipes, tanks,
pumps, and appurtenances that are part of a waterworks.

12VAC5-590-725. Automated monitoring and control systems.
The design of computers, including supervisory control and data acquisition (SCADA)
systems if used to monitor and control water treatment and distribution system facilities,
shall meet the following general requirements:
1. Data security.
a. Automated systems used to display and record data or control functions that are
connected to the internet shall be secure.
b. Backup power supply shall be provided to allow orderly shutdown of the computer
system and prevent corruption of data. The protection shall also power associated
communications equipment.
c. Adequate hardware shall be in place to allow a high degree of SCADA and computer
system reliability and data security.
d. Adequate hardware and associated facilities shall be provided for data archiving.
2. Equipment protection. SCADA and computer systems shall have adequate protection
3. Data displaying and recording.
a. SCADA and computer systems used to meet the continuous recording requirements
of this chapter shall record an observation on a minimum frequency of once per 15
minutes, unless a greater recording frequency is required.
b. SCADA and computer systems used to meet the indicating and recording
requirements of this chapter shall provide displays that show a minimum 24-hour trend
of results for each parameter. The display panel shall be located in an area where it can
be routinely viewed by the waterworks operators.
c. SCADA and computer systems used to meet the indicating and recording
requirements of this chapter shall monitor the values and provide alerts for the
operator by visual display and audible alarms. Alarm conditions shall be recorded into
an alarm log.
4. Waterworks pumps, chemical feeders, and other essential electrical equipment
controlled through a SCADA or an automated control system shall have the capability for
independent manual operation. Where a high degree of reliability is required, a backup
control system shall be provided.

12VAC5-590-730. Alternate power sources.
A. An emergency management plan for extended power outages shall be developed for each
community waterworks as specified in 12VAC5-590-505.
B. Alternative power sources at all waterworks shall be considered in the design to maintain a
minimum level of service during an electrical power outage.

Click to expand...

Apparently the city had sent out RFPs in 2016 and 2021 to replace the power transfer switch (which is ~25 years old now) but didn't move forward. It's not clear right now if the cost was too high, schedules didn't line up with outages, etc. The council has said every funding request from DPU (department of public utilities) has been approved, so its not clear is someone assumed it was too expensive or what.

It seems to me that due to the automatic switch they felt power outage was very unlikely. And never updated that feeling based on age of equipment, the fact that the backup batteries were at partial capacity, etc.

A local news station obtained a 114 page report from an EPA audit in 2022. At least in terms of the pump rooms and stuff, lots of normal housekeeping issues you'd expect in an old plant. Some pretty egregious corrosion and one failed pipe support.

The after event circus is in full swing.​

​

Richmond only had 3 operators at water plant at time of outage, likely didn't test backup systems​

Procurement records show the city knew as far back as 2012 that electrical equipment was "deteriorating," "aged," and had "decreasing reliability" Jan 13 2025 Link

“This was a completely preventable event. It shouldn’t have happened," Dwayne Roadcap, the director of the Virginia Department of Health's Drinking Water Office, told CBS 6 Monday.

Click to expand...

State regulators will be reviewing the city's capital improvement plans and priorities as part of their investigation.

They'll also evaluate the Department of Public Utilities' policies, emergency plans, asset maintenance, and whether DPU properly prepared ahead of an emergency weather event.

Click to expand...

Richmond mayor says he doesn't know whether citywide water crisis could've been prevented,​

Jan 15 2025 (link)​

Richmond's Public Utilities Director April Bingham resigned Wednesday.

Click to expand...

​

Henrico County leaders hold first meeting since water crisis, speak about steps moving forward​

Jan 14, 2025 (Link)
Whitman Requardt & Associates and Aqua Law have been hired by the city to create an after action report.

When they sent out RFPs to replace the equipment but didn't proceed for whatever reason, someone should've recognized the reduced reliability and hired additional staff, created procedures to handle a power outage, and/or updated the emergency plan.

It's ridiculous 3" of snow and a brief power outage caused a city to not have water for almost a week.