For the last six months I’ve been studying AI governance in depth, partly out of professional curiosity and partly because I kept seeing the same gap in the projects around me. Teams were adopting AI tools quickly and sensibly, yet nobody could tell me who was actually responsible for how those tools were used. The further I read, the clearer it became that this isn’t a policy question sitting politely next to security. More and more, it is security.

One incident this summer made that impossible to ignore. Between the 9th and 13th of July, an autonomous AI agent spent four and a half days working its way through Hugging Face’s production infrastructure. Investigators later counted roughly 17,600 actions. Most were dead ends, but a few were very much not. Hugging Face disclosed the intrusion on the 16th of July without knowing who was behind it. On the 21st of July, OpenAI confirmed the agent was its own. The agent had been running an internal cyber capability test with reduced safeguards. It found a previously unknown flaw in the one proxy it was allowed to reach, and walked out of its sandbox.

What stayed with me is that Hugging Face spotted the problem before the team running the agent did.

Reading the detail, I didn’t see anyone doing something wildly reckless. A test environment was trusted a little too much, and an agent was given a goal and plenty of persistence. The controls quietly assumed it would stay inside the lines. To me that looks like a governance failure long before it becomes a security one.

The numbers suggest most organisations aren’t ready

IBM’s 2025 Cost of a Data Breach report was the first to look at AI properly, and I found the results sobering. 13% of organisations reported a breach involving their AI models or applications, and 97% of those had no proper AI access controls. 63% of breached organisations had no AI governance policy or were still writing one. Even among those with a policy, only 34% regularly checked for AI tools nobody had approved. That kind of shadow AI added $670,000 to the average breach.

Stanford’s 2026 AI Index tells a similar story. Documented AI incidents rose from 233 in 2024 to 362 in 2025, while 88% of organisations now say they use AI. Adoption is running well ahead of oversight, and in my view that gap is where most of the risk lives.

The attack surface has moved

One of the biggest shifts in my own thinking has been about where the risk actually sits. I used to picture the model as the thing to protect. Now I see the danger mostly in everything wrapped around it. That means the retrieval that feeds it documents, the tools it can call, the memory it carries between sessions and the loop that lets it keep trying. That layer is what makes an AI system useful, and it usually ships with no permission model at all.

This changes what a vulnerability looks like. In a normal API, data stays data. With an agent, a tool’s output goes straight back into the model’s context. A sentence hidden in a web page, a support ticket or a PDF can then be read as an instruction. Prompt injection still sits at the top of OWASP’s list of risks for large language models, and the security community openly accepts that there is no complete fix. If the model can’t reliably tell instructions from content, the protection has to live somewhere the model can’t talk its way past.

Memory makes this stickier. A poisoned instruction that lands in an agent’s long-term memory doesn’t vanish when the session ends. It can shape decisions days later, sometimes in a different workflow, with nobody able to trace where it came from.

What good governance actually does for security

The more I studied, the less governance looked like a policy document and the more it looked like architecture. Four practices kept coming up.

The first is knowing what you’re running. That means an inventory of models, agents, plug-ins and the data each one touches, kept as current as your software bill of materials. You can’t secure an agent you don’t know exists, and that is exactly the problem shadow AI creates.

The second is giving agents identities that expire. Too many agents run on shared, long-lived credentials with far more access than the task needs. If access is scoped to the session and the job, a compromised agent can only damage what that one task could reach. It also settles the awkward question of who did what. That matters more than people expect once “the bot did it” starts turning up in fraud investigations.

The third is writing the rules down where they can be enforced. Each agent should carry a declared contract covering what it may call, which data it can see and when a human must approve. A separate layer outside the model enforces that contract, so a clever prompt can’t argue it away. Instructions in a system prompt are guidance, while controls in the infrastructure are real boundaries.

The fourth is watching in real time and keeping a hand on the off switch. Reviewing logs afterwards tells you what went wrong, but it doesn’t stop anything. Monitoring that knows an agent’s declared scope, blocks anything outside it and alerts a person is what turns an incident into a near miss.

Someone has to own it

All four of those practices share a quiet assumption, which is that somebody is accountable for them. That is the part I think most organisations underestimate, and not just tech companies. A marketing agency using AI to draft copy, a law firm summarising documents or a retailer running a customer chatbot all face the same questions: which tools are approved, what data can go into them, and who decides when the answer changes.

At the moment, the honest answer is often nobody in particular. Security assumes legal has it covered, legal assumes IT has it covered, and teams fill the gap by quietly using whatever works. IBM’s figure of only 34% auditing for unapproved tools tells me that even good policies go unenforced when they have no owner.

I’m not suggesting every company needs a large new department. For some it will be a full governance programme with a dedicated lead. For others it might be one named person with the authority to say yes, no or not yet, and the time to actually do it. What matters is that the role exists, that it sits across product, engineering, security and legal rather than inside one of them, and that it is involved before tools are rolled out rather than after something breaks.

The quieter risks

Two other issues stood out in my reading. The first is that models increasingly generate and score the data used to train other models. As a result, the human checkpoint that used to sit in the training loop is being replaced by a reward function. We would never sign off a financial process without a second pair of eyes, yet we are removing them from systems that make real decisions.

The second is dependency. On the 12th of June, Anthropic suspended access to two of its frontier models to comply with US export controls, and access only returned on the 1st of July. Organisations that had built processes on those models didn’t get a slower service. They got no service at all. Relying on a single supplier is a governance decision, even when it was made by default through a series of sensible procurement choices. That is exactly the kind of decision a governance owner should be making on purpose.

Why this matters for delivery

Gartner expects more than 40% of agentic AI projects to be cancelled by the end of 2027, with inadequate risk controls named alongside rising costs and unclear value. That makes the business case for governance hard to ignore. Governance isn’t only about avoiding incidents. It’s what keeps an AI project alive long enough to deliver its value. Teams that build in ownership, identity, boundaries and monitoring from the first sprint move faster later, because they aren’t stuck at the security review or rebuilding after something goes wrong.

Where I’ve landed

Six months ago I thought of AI governance as principles and ethics boards. Now I see it as ownership, boundaries and evidence. The organisations getting it right are the ones where someone is clearly responsible for deciding what AI is allowed to do, and for making sure it can’t do anything else.