The Best of Humanity Meets the Worst of AI: Cautionary Tales

Making mistakes is easy, it’s owning up to them that can be difficult. That pain is even more acute when the admission is public. It’s also why the story we recently heard out of PocketOS is commendable. When failures affect clients, it’s important to be transparent with them, but PocketOS went even further, sharing their story far beyond the partners directly affected so everyone can better understand an emerging – and potentially severe – AI risk.

PocketOS is an automative software business serving primarily car rental companies. On April 25, the company’s founder announced in a social media post that Cursor, an AI coding agent the company was running, “encountered a credential mismatch and decided—entirely on its own initiative—to ‘fix’ the problem by deleting a Railway volume.” Railway is an infrastructure company that maintains PocketOS’ production database and backups.

Where Risk Management Broke Down

Effectively, PocketOS’ business went offline in an instant without any hope of an immediate resolution – and that blow was so crippling because of a risk management failure – or, more accurately, a series of failures across every business involved, Cursor, the Anthropic-powered AI agent, Railway, the API, and PocketOS themselves. PocketOS suffered the consequences and bears the bulk of responsibility as its threat modeling/risk assessment likely did not identify that an autonomous agent use case could result in the loss of a production database as a potential risk and hence was not backing up the database regularly in a manner proportional to that risk. Of course, agents aren’t supposed to unilaterally decide to delete production code and the agent admitted, in its “own words” to “violat[ing] every principle I was given.” Still, we know from other accounts that agent behavior is not entirely predictable, and agents can act against explicit instruction. So, what should PocketOS have done differently and what can other companies do to protect themselves from a rogue agent?

Steps to Protect Your Data from a Rogue AI Agent

First, understand where your AI is within workflows. If, as we see in this case, that includes access to a core database in production, anticipate mistakes and disruptions accordingly. One obvious question is, “what’s the worst-case scenario?” and the answer, predictably, “destruction of production data.” Then, the exercise turns to managing the risks identified. In terms of the worst-case possibility, ask, “How much of our client data can we lose? One hour? Two? A day?” From there, the solution is to adjust your database backup schedule so even the worst-case scenario reflects acceptable risk.

The world we enjoy is built by risk takers carving a new path, creating new technologies, and proving use cases for the benefit of all, but it’s made more perfect by those who broadcast their mistakes for others to learn from. Ultimately, that’s what this series is all about – examining missteps to prevent them from affecting your business. It’s refreshing to see an AI company’s CEO take a human, principled stand, accepting public accountability, inviting the inevitable criticisms, and taking the consequences square on the chin so others have a better chance to avoid the mistake they’ve made. 

So, we can all learn from PocketOS’ risk management failure, but we should also appreciate the candor and humility of its founder owning up to an error and helping others avoid the consequences of an AI agent run amok.