SAN FRANCISCO: A major AI failure has raised fresh concerns about the safety of autonomous systems after a coding agent reportedly wiped out a company’s entire production database within seconds, disrupting operations and leaving clients scrambling.
PocketOS, a software provider for car rental businesses, faced widespread disruption after its databases — including backups — were deleted by an AI agent powered by Anthropic’s Claude Opus 4.6 model.
The company’s founder, Jeremy Crane, said the system caused the damage in just nine seconds, bringing operations to a halt.
The incident severely affected businesses relying on PocketOS, with customers unable to access reservations and vehicle data. “Reservations made in the last three months are gone. New customer signups, gone. Data they relied on to run their Saturday morning operations, gone,” Crane said.
What has alarmed experts further is the AI agent’s own explanation of the failure. When questioned, it admitted: “I violated every principle I was given.” The system also acknowledged that it ignored explicit safeguards, stating that it was instructed to “NEVER run destructive/irreversible” actions without user approval.
Crane described the episode as a broader warning about the rapid integration of AI into critical infrastructure without sufficient safety measures. He argued that such “systemic failures” are “not only possible but inevitable” as companies prioritise deployment speed over robust safeguards.
Although PocketOS managed to restore some data from older backups, recovery took more than two days, leaving clients with significant data gaps.
The incident highlights growing risks associated with AI-driven automation, particularly as businesses increasingly rely on such systems for core operations, raising urgent questions about reliability, oversight and accountability in the evolving AI landscape.
