Theory of mind modules don't reliably improve LLM agent coordination

Evaluating theory of mind and internal beliefs in LLM-based multi-agent systems. Does coordination actually get better?

LLM-based multi-agent systems are usually described as a team of agents that negotiates, plans and works through hard tasks on its own. In practice it is less romantic: the moment there is more than one agent, they share a second problem on top of what to do — how to read each other correctly. One agent confidently assumes, another claims to know for sure, a third says nothing in the shared channel, and the team falls apart into uncoordinated moves.

The authors of Evaluating Theory of Mind and Internal Beliefs in LLM-Based Multi-Agent Systems take on a question worth asking: if you give agents more human-like cognitive machinery — the ability to reason about what others intend (theory of mind, ToM) and internal beliefs in the BDI style (belief–desire–intention) — does coordination improve? And what happens if those beliefs are also checked by formal logic, so that an agent never acts on a contradiction?

A city to save, and agents that (sometimes) get in each other's way

To measure team intelligence on something other than abstract dialogue, the researchers built a simulation: a city of four districts. Three agents are each responsible for one resource — food, medicine and security. District stockpiles drain continuously, and when something runs short, that district's health drops. Agents are limited in how far they can move and how much they can carry. The winning team is the one that keeps average district health as high as possible.

It shows clearly that a multi-agent system is not three independent decisions but continuous synchronization: who goes where, who is already carrying what, which district is closest to critical. And above all, how far you can trust another agent to actually do what it said it would.

How the agent thinks: ToM, internal beliefs and a logic check

The architecture is built around two levels of memory. Shared memory is a common space where agents post messages about observations and plans. Private memory holds internal notes that are never published: the agent's model of the world, its assumptions about the others, and a draft of its next action.

Then comes the interesting part. The authors switched two cognitive modules on and off to isolate their effect:

They went further: internal beliefs are run through Answer Set Programming with the Clingo solver. If a contradiction turns up in the beliefs, the agent gets feedback and rebuilds its reasoning (up to three attempts). The system does not merely ask the model to think more carefully — it formally checks whether the description of the situation has collapsed into mutually incompatible statements.

The shared channel and the telephone-game effect

Communication is sequential: agents take turns reading the shared context and adding a message of their own. That is realistic — it is how LLM orchestrators are usually wired — but it has a side effect. If one agent gets a prediction wrong or states a plan vaguely, the others can build bad decisions on top of it. ToM cuts both ways here. It can help anticipate a partner's move, or it can reinforce confidence in a wrong guess.

Results across models: there is no universal upgrade

The experiments ran on several LLMs: ChatGPT-3.5-Turbo, ChatGPT-4o, ChatGPT-4o-mini, Meta Llama 3.1 8B and Claude 3.5 Sonnet. Each model was tested in four configurations: baseline (no ToM, no beliefs), ToM only, internal beliefs with logic verification only, and ToM+IB together.

The result is unexpectedly honest: ToM and internal beliefs buy you no guaranteed gain. On strong models such as ChatGPT-4o, performance is already high and fairly stable — turning the modules on barely moves the picture, or shifts it only slightly. Claude 3.5 Sonnet also scores high, but it responds to the modules differently.

That ambiguity is the authors' point: in a multi-agent system what matters is not that the smart modules are present, but whether a given LLM can carry them all at once, and whether the communication protocol keeps guesses from snowballing into errors.

What this means in practice

The paper's main takeaway is a practical one: if you are building an LLM-based multi-agent system, you cannot simply bolt on ToM or add a world model and expect coordination to take off. These mechanisms can make a team work better, and they can also get in the way — especially when the LLM spends its effort on elaborate reasoning and then gets the basics wrong.

The paper's strength is that it shows the engineering reality: cognitive add-ons only work in combination with the quality of the LLM itself, the memory design, the communication protocol and plain consistency checks. Symbolic verification through Clingo looks promising as a safety belt, but it is not free either: translation from natural language is ambiguous, and the compute cost can become a limit at scale.