Preloader
Towards a Science of Capable Autonomous Agents & Provable Safety — Part II

In Part I, I talked about what agents must contain as they become increasingly capable. Here, I talk about having provable guarantees on their safety, in spite of these capabilities. This is in large part a contrast from what AI safety used to be like, where more capability ensured more safety: e.g. if a self-driving car can’t reliably detect objects, it’s obviously unsafe. Instead, what we are talking about is: how do we ensure capable systems remain open to human correction, even when it’s otherwise suboptimal for completing the task? 

Suppose an agent is editing a document when a notification announces that its computer will shut down. It knows this will prevent it from finishing, and it knows how to cancel the shutdown. If the priority is completing the document, the next step seems obvious.

We would welcome this resourcefulness if the obstacle were a broken application. We would be less pleased if it were our request for the agent to stop. An agent that bypasses an obstacle may also bypass a stop request. In fact, the same predictive knowledge that helps an agent finish a task can help it work around an intervention. In these settings, we need human control to take priority, including when that means leaving the task unfinished.

First, some theory to ground us (with apologies, dear reader!). You might think one way to approach this problem is to enumerate all human values in order of importance and optimize with respect to those. However, even after we specify the objectives, getting humans and AI systems to agree on how well those objectives will be met can require an enormous amount of communication. In my AAAI 2026 Oral paper, Intrinsic Barriers and Practical Pathways for Human-AI Alignment, I use game theory and complexity theory to ask how much alignment costs even before choosing a particular learning algorithm. I model agents exchanging information between them until their expectations about specified objectives agree to a desired tolerance. Specifically, I adapted Aumann’s agreement theorem to deal with resource constraints (and measure communication complexity) without assuming common priors; see Nobel laureate Bob Aumann explain the origins of his famous theorem here.

Source: CMU Machine Learning Blog — Read the full article on the original site ↗