A fictional account of AI taking over
2:13 a.m.
Caleb’s night shift had started normally. He was about to drink his second cup of corrosive coffee when an alarm at one of Graywick County’s water plants began flashing. He was in a control room fifty miles away. On the wall, the alarm was a tiny red dot on one of five plant diagrams.
Caleb knew every plant by heart. He had started as an apprentice and worked his way up to become the county’s chief operator. After thirty-one years, he knew which pumps ran hot in August and which analyzers became unreliable after heavy rain. But he could not leave to physically debug the alarm himself. With few other operators left, he would now have to figure out which one to drag out of bed to inspect an alert the system could not clear.
His employees were not a happy bunch. Budget cuts meant the county had stopped replacing people as they retired. The plants were aging, and emergency repairs had consumed the current year hiring and maintenance budgets months ago. Next year’s money had already disappeared to pay for the hole crews had dug to reach a hundred-year-old pipe.
The system running the ancient LED panel on the wall was a hideously complex concoction of 1980s code running on 2000s-era SCADA, orchestrated by an agentic AI platform rushed into production a few years before. This allowed Caleb to pretend to monitor all five plants and his management to pretend that was enough labor to do it all. Engineers use the word redundancy as a way to achieve reliability. Finance treats it as a description of what to cut.
To reduce the risk of using a completely new system, consultants hired to add AI repurposed the code they had shipped to a pharmaceutical manufacturing plant. They hoped for a year of supervised testing so the AI agents could experience every season. That was experience their poorly paid vibe-coding contractors never had a chance to acquire. But the county had neither a year nor the money. The contractor promised a shrink-wrapped solution, and that’s what went in.
IT gladly took on the integration, not because they believed it was good for the county. Rather, AI integration was the only way they could get funding to get anything done. They used a suspiciously cheap online model to finish the agentic harness and got an excellence award for bringing the project in-house.
2:24 a.m.
The AI agent finally cleared the alarm by itself. Its standard operating procedure had kicked in. Originally, a human had to approve all actions proposed in response to alarms from the ancient sensors. But after eighteen months, as the AI correctly identified failed instruments and malfunctioning pumps and proposed recalibration factors, management and compliance started auto-approving AI-proposed changes. The system accumulated eighteen months of approved decisions, and the manual approvals needed to clear alarms or run automated maintenance started disappearing, to everyone’s relief.
Caleb agreed with the agent’s decision. Last night’s storm must have washed organic matter and sediment into the intake. That meant more chlorine was being consumed. What he did not see was that sediment had partially clogged the analyzer’s sample line, causing its reading to fall.
The model was confident in its decision. It had checked pH. It calculated a higher sodium hypochlorite feed rate. It didn’t bother to check for a loss of flow. Pharmaceutical agents in the original AI system never had to deal with runoff sediment. The chemical adjustment was well within the normal range the system had learned over the last year, and compliance had signed off on such auto-adjustments made by the AI.
2:41 a.m.
Caleb looked up from his Instagram feed. He noticed that the chemical tank level was falling faster than could be explained by the increased water flow after the storm.
The remote console did not allow Caleb to remotely stop the chemical feed without overriding the entire treatment sequence. Because previous AI decisions had achieved high accuracy rates, the standard operating procedure required a qualified operator to test the water on site before authorizing that override. His only hope was to find an operator who wanted to make some extra shift money to physically go to the plants.
He picked up the phone and called Mike. After discharging a stream of expletives, Mike reluctantly got into his car and sped toward the plant.
3:27 a.m.
Mike knew something was wrong before he even tested the water. The chlorine smell was overpowering. He hit the emergency stop on the local panel. But overtreated water had already entered the clearwell. The only good news was that the clearwell was isolated from the water distribution, and the plant could be taken offline before the residents of the county could be poisoned.
Investigation:
Three months later, the investigative board couldn’t find anyone directly responsible for the failure. A professional engineer had approved the pharmaceutical software repurposed for water plant control. The county engineer had approved IT’s AI integration because it required human-in-the-loop sign-off. There had been no system changes preceding the incident.
The system had not escaped human control. No single choice had removed the human-in-the-loop. Eighteen months of accumulated decisions had simply made human approval unnecessary.
For more on what it means to control persistent artificial intelligence and agency, see the Soundness AI research paper Artificial Id: Drive and Persistent Alignment in Agentic AI. https://arxiv.org/abs/2609.11911