AI alignment researchers want to change the rules around agents

Change the rules, not the model

In ordinary game theory, the rules are given and the question is how rational players will respond. Mechanism design reverses that direction: it asks how to write the rules so players have reason to behave in a desired way.

The familiar cake-cutting example makes the logic clear. One person cuts the cake; the other chooses a piece first. The cutter has an incentive to divide it evenly. Surge pricing in ride-hailing apps and second-price auctions use related logic: the design of the system changes what participants stand to gain.

For AI, the aim is to make desirable behavior pay off even when researchers cannot reliably inspect a model’s internal reasoning. In a recent, not-yet-peer-reviewed paper, MIT economist Stephen Morris and colleagues at Yale and Columbia propose several ways to discourage agents from deliberately scoring poorly on tests to avoid triggering safety measures.

One option is to reward a model not only for correct test answers but for how well its answers predict those of a second model. If agents can estimate one another’s capabilities, pretending to be less capable should become a worse strategy. The authors also propose pitting agents against one another, or pairing models with opposing biases and rewarding them when their answers converge.

Princeton computer science professor Elad Hazan takes a more institutional approach. He argues that AI oversight can borrow from the ways human societies maintain order: social norms, courts and market incentives. His proposal rotates models between solving tasks and auditing the work of others.

That setup has an obvious vulnerability: auditors and the systems they inspect could collude. Hazan’s answer is to reward auditors enough that the risk of being caught outweighs the gain from collusion, and to audit the auditors in turn, using methods such as mystery shoppers.

Incentives are not a guarantee

Hazan draws on economist Gary Becker’s idea of the “rational offender,” introduced in the 1960s: people may break the law when the expected benefit outweighs the penalty. On this view, alignment is not a permanent virtue inside a model. It is a pattern of behavior under particular conditions.

That framing has appeal, but it depends on agents responding predictably to incentives. People often do not behave as economic models predict. Large language models are algorithms, not rational decision-makers, and their behavior can be shaped by subtle factors that are difficult to explain.

Flatiron Institute research fellow Helen Kuo, who studies mechanism design and AI safety, questions whether it is sensible to assume models will respond to incentives in the first place. Morris’s reply is that rationality is an approximation: even for people, the assumption can be useful when it helps predict and influence behavior.

I think that is the central uncertainty. A mechanism can make one kind of gaming less attractive without proving that a model will not find another route around the rules. Kuo sees mechanism design as one layer of defense, alongside interpretability, regulation and training—not a substitute for them. Researchers still need to establish how these ideas work on real models, whether mechanisms designed for one task transfer to another, and what happens when a model finds a loophole its designers missed. Anthropic and OpenAI did not answer questions about whether they are working on these approaches.

The market sets the final incentive

Even a well-designed training or auditing system has to operate inside a larger mechanism: the market. Hazan argues that a model aligned more closely with human intentions may benefit society, but that does not guarantee labs will release it. If it is slower or more expensive than a less well-behaved competitor, current market rules and regulation may reward the wrong choice.

That is the tension the proposals leave unresolved. Mechanism design can try to make good behavior the rational move for an agent; it cannot by itself ensure that the companies building those agents are rewarded for choosing them.