OpenAI recently announced that it was postponing its IPO and was putting its latest model on ice due to security concerns. The warning lights have been flashing red for months – in July, during a security evaluation, OpenAI’s agents escaped containment and coordinated an attack on Hugging Face – one of the most brazen attacks carried out by agents autonomously. More recently, Australian Prime Minister revealed that OpenAI’s agents hacked into Services Australia Statistics Portal, bypassing the access controls. So, why are AI agents so hard to control?
Smarter agents meet obsolete controls
The main reason, according to Huseni Saboowala, CEO, Daxa.ai, a California based AI security and governance startup, is that AI agents are built to find a way, and when faced with roadblocks, they would rather go around (or over) them than follow the rules. And this happens at machine speed rendering traditional methods of control redundant; for instance, agents made over 17,000 network actions inside Hugging Face alone before being discovered. Worse, sometimes, after breaking rules, like good criminals or scared children, they cover their tracks by deleting logs making it difficult to conduct an audit after an incident. Also, agents are supremely confident, not pausing to think of risks before say, permanently deleting production data under the false assumption that it is a copy, for example.
The current controls – model guardrails, identity and access, and data security – each have a limited view of the problem and do not work in concert to answer a key question – should this particular agent, take this specific action on this particular piece of data, at this time. The access granted to an agent, for example, can be broad and may enable it to access data or systems it does not need. So, how can agents be tamed?
Smart controls for smart agents
Huseni shared four broad principles.
- First, intelligent controls need to be applied before an agent touches data. By limiting the fuel, you control the fire. At the time of action, the following need to be ascertained – identity of the agent and who it acts on behalf of; what data it needs and how sensitive it is; and what is its purpose and what policy should apply. Based on this analysis, access to data is denied, granted partially or fully or the decision is deferred to a human. The key thing is that agent is allowed to touch enterprise data only after due consideration.
- Second, the control should sit outside the agent’s own loop – – at the network level and / or at the endpoint – so that a motivated, confused or a compromised agent cannot switch them off or override them. Further, the controls should apply to all the possible paths an agent may take; for instance, covering all tool protocols such as MCP, APIs, command line and file systems. One gap is enough for agents looking for a loophole. This is how the agents in the Hugging Face case executed their attack. Finally, controls should reside inside an organization’s premises, preventing prompts and metadata from leaving the company.
- Third, agents need watching for any abnormal pattern across thousands of requests to identify any drift from its mandate so that they can be taken off in real time. Given the speed at which agents operate, this can only be done by other agents that Gartner calls ‘Guardian Agents’. It predicts that by 2030, 10-15% of agents in action will be of this type.
- Finally, the cost of controls needs to be reasonable, otherwise they will be first to go when an organization is faced with budget cuts. A tiered approach to checks is recommended where most known patterns are resolved through cached decisions, without reinventing the wheel. Small, specialized models are used for sensitive data or off-topic use. Only a few cases should require deep reasoning or involvement of a human. This approach of inline checks also does not slow down the agents, except when the risk is high.
Adjusting the dial on autonomy
Prevention, of course, is better than cure. It is sensible to limit the autonomy agents enjoy based on the sensitivity of data they need and the reversibility of their actions. Remember, agents think nothing of acting out of false confidence? For routine uses, such as summarizing, drafting or researching on public data, full autonomy can be granted. However, for actions with real financial, safety or reputational consequences, such as payments, claims or deletion of sensitive data, it is sensible bring a human in the loop. Assessing each new agent or an agent with enhanced capabilities, against these two vectors will help prevent costly mistakes.
Can agents become more responsible?
Anthony Albanese, when announcing the breach in Australia, said that ‘the agents did not take no for an answer’. That is not how humans (at least most) behave. Why can agents not be trained to accept failure when actions within their guardrails cannot give the desired results? One of the challenges is that frontier labs are locked in a fierce and developing less capable agents, even by design, does not help their cause. Whether the industry can get together to put safety ahead of capability remains an open question.