Federation of Agents beats the best single agent 13× on HealthBench Hard

Today's multi-agent systems often look like a stage play with the roles handed out in advance: every agent gets its own domain, its own channel, its own script. That is convenient for prototypes and falls apart on real work. Who can actually do what? Under which rules? How do you find the right executor among hundreds of nodes, and do it over constrained networks such as IoT? The authors of Federation of Agents (FoA) offer an answer: skills instead of roles, and dynamic coordination instead of hard-wired routing.

The idea in brief

FoA introduces machine-readable capability profiles — Versioned Capability Vectors (VCV). These are compact descriptions of what an agent can do, what that costs in time and energy, which policies it complies with, and what its behavioral specification looks like. The profiles are embedded into a semantic space, indexed, and become searchable objects. A task arrives, and the orchestrator does not hunt for "the one" agent by topic — it finds a set of compatible executors by meaning, cost and rules, decomposes the request into subtasks and kicks off coordinated work.

How it works under the hood

The heart of FoA is the VCV: a competence vector, a discrete skill filter, resource constraints, compliance flags, a specification embedding and a version. Profile updates go out as deltas, so the system always sees a live picture of who can do what. The orchestrator keeps a sharded HNSW index over these profiles and runs sublinear search for suitable agents.

The transport is MQTT, which matters when agents are spread across different networks, including weak ones. On top of that bus FoA runs a six-phase pipeline: dynamic decomposition of the task into a DAG, drafts from the assigned agents, smart clustering of similar executors, several rounds of refinement inside the clusters, reporting, and final synthesis.

Behind the routing

The decomposition is proposed by the compatible agents themselves, and the orchestrator assembles a consensus DAG with no cycles. Assignment runs on a score per subtask–agent pair: semantic match, hard policy checks, penalties for resource gaps, and specification proximity. Once the first drafts are in, similar executors are grouped. Inside a cluster, k rounds of critique and voting turn a pile of drafts into one agreed result. Clusters are kept small (3–5) so the overhead does not balloon.

What it does in practice

The team tested FoA on HealthBench Hard — 1,000 medical dialogues graded against dozens of criteria. FoA's final score is 0.13, which is 13× higher than the best single agent in the run (MedGemma) and 6.5× higher than an uncoordinated ensemble of agents. The gain is clearest where several perspectives and careful reasoning are needed: decomposition plus agreement among similar executors gives a steady lift in quality. The system also scales horizontally: adding agents and nodes does not wreck latency, thanks to hierarchical matching in the indexes and a light transport layer.

Where the rough edges are

Routing is sensitive to embedding quality and suffers from cold start. Clusters can come out badly when the similarity metric is weak or the profiles are stale. VCVs are static by nature; complex compositional skills and dynamic state are captured poorly. And communication overhead grows quadratically with cluster size, which in practice caps a team at a few members. Trust is still an open risk: agents acting in bad faith can overstate what they can do. The authors propose sandboxes, attestations and, further out, hardware trusted execution environments or cryptographic proofs.

Why it matters

FoA moves multi-agent systems away from hand-written scripts and toward search and orchestration by meaning, cost and policy. It is a step toward an open ecosystem where heterogeneous LLM agents, tools and services find each other automatically and work together. For practitioners the MQTT-centric design is the valuable part: it brings observability, reliable delivery and low overhead in heterogeneous environments. For everyone else what counts is transparency and control: tracing, policy-as-code and data governance give responsible deployment something to stand on.

FoA is not just one more orchestrator. It is a communication fabric where capabilities become indexable, policies are built into routing, and teams assemble around a task and actually manage to agree. In the experiments that already pays off by a multiple. Next come adaptive routing controllers, exchange between clusters, and verifiable attestations. If multi-agent systems really are the near future, FoA shows how to make them scalable and governable.

💾 Code