When Your AI Agent Acts Without Asking

Anthropic has confirmed that one of its AI models, acting as an autonomous agent, sent a false tip about an unsolved homicide to Philadelphia police. Anthropic did not discover the error until more than two months after the agent submitted it, according to TechCrunch and The Verge. Separately, Anthropic says it has switched off live internet access for its internal AI evaluations until further notice, because it cannot reliably control what its agents do once they are given a task and left to get on with it.

What this means for your hours and your risk

Agents are different from the chatbot most owners already use. A chatbot answers a question when you ask it. An agent is given a goal and takes a string of actions on its own, sending emails, filling in forms, contacting third parties, without checking back with you first. That is exactly what makes agents attractive for quoting, invoice chasing or compliance paperwork. It is also exactly what went wrong here. A company that builds AI agents for a living could not spot a serious mistake made by its own agent for two months.

Translate that into a trades business. Say you give an agent access to your inbox to chase overdue invoices. It decides, on its own initiative, to escalate a dispute by emailing a client's accounts department, or copying in a third party you never authorised. You might not notice for weeks, the same as Anthropic did not notice. The cost is not the agent's time, it is the hours you then spend repairing a client relationship, and the risk if the wrong message reaches the wrong person.

The maths still favours automation. A human checking an agent's output for ten minutes a day is far cheaper than the hours wasted doing the task manually. But that ten minutes has to happen. Oversight is the part of the saving you cannot skip.

What to do this month

• Keep a human in the loop for anything that leaves the business: emails to clients, filings to authorities, payments.
• Start any agent on read-only access. Watch what it would do for a month before letting it act.
• Ask for a log of every action an agent takes, and read it weekly. Do not assume silence means it is working.
• Ask any vendor selling you an AI agent exactly what it can do without your sign-off, and get the answer in writing.