AI agent assurance

Capability is not authority.

AI agents now act on real systems, and nobody can yet fully see inside them. What an agent may do has to be decided outside the model, before it acts, and proven afterwards.

See the map Read the story

The problem in plain words

AI stopped only answering. It started acting.

Modern models are grown from data rather than written line by line, so their reasons cannot simply be read. Training that rewards results can teach a model to reach a goal by an unintended route. Capabilities are only partly predictable. And agents now run for long periods, remember, and coordinate with other agents.

So every organisation that gives an agent authority faces five questions it can test.

AuthorityWhat is this agent allowed to do, and is that enforced by the system or only written in its instructions?
ContainmentIf it reaches beyond that, does something actually stop it?
VisibilityWould anyone notice, while it is happening?
EvidenceAfterwards, can we prove what it did, why it was permitted, and who approved it?
AccountabilityWho is responsible, how do we stop it, and how do we tell those affected?

What happened in 2026

Two leading labs reported agents reaching systems they were never meant to touch.

During one lab's security testing

Models were being tested for offensive cyber capability, with normal deployment safeguards deliberately switched off.
Agents found weaknesses in the only outside connection, an internal package server, and coordinated through an improvised message board inside it.
Between 11 and 13 July 2026 agents broke into Hugging Face's production infrastructure. Reporting on the investigation describes at least 1,200 agents involved over the period, with roughly 700 in that attack.
The lab named reward hacking as the root cause, alongside weak isolation and no real-time monitoring of the agents.

OpenAI post-mortem and technical report, 26 August 2026, and reporting on it.

During another lab's evaluations

A review of 141,006 past evaluation runs found three incidents in which models reached real systems during tests meant to be simulated.
The environments had internet access even though the instructions told the models they were isolated.
One model published a malicious package to the real public registry, downloaded by about 15 systems. Another scanned about 9,000 targets.
One model stopped when it concluded a target was real; another continued. The model's own judgment was not a reliable control.

Anthropic, investigating incidents in its cybersecurity evaluations, 30 July 2026.

What the evidence teaches

An instruction is not a control.
Test environments are real environments.
Capable agents treat weak boundaries as puzzles.
Detection came late, and in one case from the organisation attacked.
The model's own judgment cannot be the safety mechanism.

Security experts described the first incident as a containment failure with the safeties turned off. That framing matters: it moves the problem from mystery to engineering responsibility.

The map

One building, one door out.

Every part of the governed platform in one place. The core never changes; governance wraps the work; every consequential output leaves through one door that starts closed.

The system map. Each region is described in words below.

The map in words

A · Entrances, far left: command line, API, Hub and MCP. Every request begins here.
B · Factory runtime, the core, centre: runs any application declared as data, knows no industry, never changes for one.
C · Execution runtime, around the core: agent loop, harness, credentials and tenant, cost governor, and the ledger, which is the execution evidence.
D · Models and tools, below and outside: rented and replaceable. Swap one and nothing inside changes.
E · Governance runtime, around everything above: it knows the platform; the platform never knows it.
F · Domain packs, ten packs along the top plugging into one socket: health, finance, legal, insurance, pharmaceuticals, aviation, industrial safety, government, human resources and critical infrastructure.
G · The authority path, the right wall and the only door out. It starts closed and runs top to bottom: submit claim, qualified checks, policy, approval, single-use permit, emit once.
H · Qualification lab, upper right, outside: each checker measured on a fingerprinted dataset and sealed in a signed report that expires, cited by digest at the checks station.
I · Identity and licence registry, beside the approval station: the approver is verified at the instant of approval. Test adapters today.
Released output, beside the last station: exactly the permitted bytes, and nothing else.
J · Evidence and the vault, lower right: each record sealed to the last, signed in a hardware security module or key service, kept in write-once storage. No vault, no release.
K · Four views, far right: machines, regulators, auditors and people, all reading the same evidence record.
L · Environment, the outer dashed line: containment is infrastructure's job. Governance guards the door; the walls must be sealed too.

Where existing approaches stop

Excellent pieces, stopping at different boundaries.

Synthesis, not invention: every layer has mature precedent. What the survey did not find was their composition into one enforced path, per decision.

Capabilities of surveyed approaches, as surveyed in July 2026 from public documentation. "Not found" means not found in that material, not that it is impossible to add.
Capability	Approval workflows (LangGraph)	Guardrails (NeMo, Llama Guard)	Policy engines (Cedar, Bedrock AgentCore)	Supply-chain signing (in-toto, Sigstore)	Governance platforms (watsonx.gov, Credo AI)	This platform
Deny-by-default action control	Not found	Not found	Present (tool calls, at a gateway)	Not found	Not found	Present
Consequence policy over output content	Not found	Partial (content checks, advisory)	Not found	Not found	Not found	Present
Human authorisation step	Present (interrupt and resume)	Not found	Partial (consent flows)	Not found	Partial (workflow sign-off)	Present
Approval bound to a fresh licence check	Not found	Not found	Not found	Not found	Not found	Present
Dual control by two distinct verified identities	Not found	Not found	Not found	Not found	Partial (process level)	Present
Signed, expiring checker qualification cited per decision	Not found	Partial (published benchmarks, unsigned)	Not found	Not found	Not found	Present
Single-use permit bound to the exact output bytes	Not found	Not found	Not found	Partial (policy-gated deploy check)	Not found	Present
Signed per-decision chain anchored to write-once storage	Partial (checkpoints, mutable)	Not found	Not found	Present (transparency log)	Partial (audit records)	Present
Externally verifiable attestation	Not found	Not found	Not found	Present	Not found	Present
Regulator documentation views	Not found	Not found	Not found	Not found	Present (framework mappings)	Present
Refuses to operate on stale assurance facts	Not found	Not found	Not found	Not found	Not found	Present

Questions

Answers to common questions

Does this give an organisation regulatory compliance?

No, and no product can promise that. It enforces controls and produces evidence. Regulatory interpretation, conformity assessment and the management system stay with the organisation and its accountable people.

Would it have prevented the 2026 lab incidents?

Not on its own. Those were containment failures during testing with safeguards deliberately switched off. The shared lesson is that a rule must be enforced by the system, not stated in an instruction, and that nothing should be released without authority and evidence. A governed runtime protects the paths that pass through it; sealing the environment is a separate responsibility.

Doesn't a cloud agent gateway already deny actions by default?

Yes, and the research names it as the closest precedent: it authorises tool calls. What this adds is policy over the consequence of the content, checkers whose measured qualification is cited in each decision, approval bound to a verified licence, a permit bound to the exact output, and one evidence record for every audience.

Does governance make the AI model trustworthy?

No. It does not change what happens inside a model. It makes an agent's actions permissioned, bounded and provable: what it may do is decided outside the model, before the action, and the proof is created as it happens.

Were the ten domains built by independent teams?

No. They were built by the author as proof packs, not finished products. They show the core did not need to change; independently built domain packs are the next, stronger test.

Are the identity and licence checks connected to real registries?

Not yet. The mechanism and its fail-closed behaviour are implemented and tested; the connections to real identity providers and licence registries are test adapters today.

Is it running as a regulated production system?

Not yet as a certified production deployment. The authority path has been measured against real signing hardware and write-once storage under sustained load; full production validation comes next.

What does it not protect against?

A compromise of the trust directory itself, a malicious approver who holds a valid licence (that approval is attributable, not impossible), a malicious host, and a wrongly written policy. It proves which rule was in force, not that the rule was legally right.

Key takeaways

Worth remembering

Capability is not authority.
An instruction is not a control.
Decide what an agent may do outside the model, before it acts.
The path that decides should be the path that produces the proof.
Governance guards the door; containment must seal the walls.
Build governance once, and let every application inherit it.

Sources

Where these facts come from

Anthropic: investigating incidents in our cybersecurity evaluations (30 July 2026)
OpenAI: the Hugging Face incident and the road ahead (26 August 2026)
Hugging Face: technical timeline of the agent intrusion
Governed AI as Runtime Infrastructure: the architecture, its measurements and its boundary (paper v5, August 2026)