AI is not your friend

Jer Crane is the founder of PocketOS.

He is also someone who has no idea how LLMs work and should not be allowed near a production environment until he learns.

Instead of focusing on the (many) errors in judgment, I instead look at how basic AI governance would have prevented this comedy of errors.

The whole 1000?
The whole 1000?

I am not going to recap the entire lengthy X post; you can read it yourself if you want a good laugh.

An AI Agent Just Destroyed Our Production Data. It Confessed in Writing.

I originally saw this on HN, and by the time I finished reading, it was already being eviscerated. There are so many red lights, but those who have to deal with the fallout have been seeing a nearly exponential increase in exactly these types of situations.

The “I” here is Artificial and nonexistent
The “I” here is Artificial and nonexistent

This, right here, is the core LLM limitation that Jer doesn’t (or refuses to) understand. An LLM can never answer why it did something correctly. It can only give you ONE of an unlimited number of ways it COULD have done so. The LLM is hallucinating the entire answer starting with “and that’s exactly what I did”. No. It’s “exactly what I COULD have done”.

Where AI Governance would have helped

If I was limited to a single policy to implement it would be:

AI systems must never have unilateral authority to perform destructive, irreversible, or production-impacting actions.

Any such action must be blocked by a technical control that requires explicit human approval through a separate trusted mechanism.

This policy applies to actions such as:

  • Deleting or overwriting data
  • Dropping databases or storage volumes
  • Modifying production infrastructure
  • Revoking access or credentials
  • Deploying code to production
  • Changing security controls
  • Initiating financial transactions
  • Sending externally binding communications

Under this policy, the failure would have been prevented at several independent points:

The agent’s token would not have had production deletion rights. Staging credentials could not operate on production resources. A destructive API request would have required separate human authorization. Backups would have existed outside the affected administrative and storage boundary. The action would have failed safely even if the AI ignored every instruction it had been given.

The core principle is:

Treat the AI as a potentially fallible operator, not as a trusted authorization boundary.

Now let’s replace the word fallible with adversarial. No need for mentioning boundaries.

Treat the AI as a potentially adversarial operator

A human may delegate analysis, preparation, and recommendations to AI. A human may not delegate final, irreversible authority where a single incorrect action could cause material harm.

Writing a complete AI Use Policy

When starting to tailor a complete AI Use Policy I would start with:

A human must always retain the ability to understand, challenge, and override any material AI-driven decision or action before irreversible harm occurs.

That sounds broad, but it collapses many good AI governance principles into a single rule.

Why this one?

Because nearly every AI failure ultimately comes from ceding authority without preserving agency. Whether the problem is hallucinations, bias, prompt injection, automation mistakes, security issues, or unexpected behavior, the common failure is that humans can no longer effectively intervene. This is why the ability to steer was such a major improvement to the loop.

This single policy implies:

  • Transparency. People must know AI was involved (throughout the lifecycle).
  • Explainability. There must be enough reasoning or evidence for someone to evaluate it.
  • Reversibility. If something goes wrong, it can be undone.
  • Auditability. Actions and prompts should be logged.
  • Approval gates. High-impact actions require human confirmation.
  • Kill switch. AI can be stopped immediately.
  • No blind trust. Humans remain accountable for important outcomes.

This policy also scales across different domains:

AI wrote some code -> Developer reviews it before merging. AI dropped your table -> table can be restored from backup or log. AI controls OT infrastructure -> Operator can override automation.

This can be expanded as little or as much as necessary.

Post-Post-Post Mortem

Hint: the “backup” was purely accidental or a byproduct of some other process.
Hint: the “backup” was purely accidental or a byproduct of some other process.

I am glad the customers are operational and hopefully PocketOS can recover, but I urge Jer to seek expert help beyond legal counsel.

Data Privacy, Information Security, and Cyber Risk Management would all have required controls that would have certainly alleviated the risks.

Tags: