Matrix drops the central orchestrator and scales agents 15x

Synthetic data generation today runs on several agents at once — one writes text, another scores it, another calls tools, another picks the best candidate. High-quality data requires agents that can interact with each other and with their environment, producing long, branching scenarios with multiple dialogues that return only the best variant. All of this makes it hard to scale an agent to thousands of parallel scenarios.

In existing pipelines, a central orchestrator usually owns the dialogue history and hands out assignments. In practice that node quickly becomes the bottleneck: a large volume of parallel tasks puts heavy load on the network and the CPU, which leaves GPUs idle and throughput low.

Matrix attacks the problem by taking the orchestrator out of the scheduling loop and storing task state directly in the messages. Agents read those messages, do their work, and forward the messages on to other agents. Tools and LLMs move out into external services, which makes the agents themselves light and cheap to run.

What's new here

• Peer-to-peer orchestration. A message carries both the data and the control. Agents are stateless: they hold no internal memory and scale like ordinary processes.
• Row-level asynchrony. Scheduling happens per item instead of per batch. That removes the bubbles where one long case holds up an entire batch, and GPU utilization goes up.
• Message offloading. Large history payloads go into the Ray Object Store, and only IDs travel over the network. That saves bandwidth and prepares the platform for multimodality.
• Offloading heavy operations. LLM inference and containers run as services (Ray Serve, vLLM and SGLang) over gRPC with a local cache of replica addresses — less overhead and less traffic on the head node.
• Flexible assembly. Agent roles, input-output schemas and resources are declared through Hydra; monitoring runs in Grafana.

To see the contrast, look at the diagrams: the classic central-node design on top, the Matrix P2P approach below.

How it works in practice

1. Collaborative reasoning. This test uses a multi-agent pipeline in which two agents disagree with each other and then converge, generating hard scenarios for training a multi-agent task grader. The comparison is against Coral, and Matrix preserves the quality of the agreement, removes the scheduling bottleneck and scales further. On 31 A100 GPUs it sustains 12,400 concurrent tasks at 129,800 tokens per second overall, against 18,900 for Coral, with a peak gain of 6.8×, while holding a comparable level of agreed-on correctness.

2. Question-answer extraction. This test is built on the NaturalReasoning benchmark, which mines hard questions out of 25 million web pages. After filtering, it collected 1.19 million examples. It involves three agents: a classifier on LLaMA-3.1-3B for a fast check of whether a document is usable, plus a generator and a grader on LLaMA-3.1-70B that produce the questions and verify they are useful.

3. Support for communication scenarios (Tau2-Bench). Here the agents hold a dialogue with a user, call tools over HTTP and containers, and get an honest score. The baseline implementation topped out at roughly 500 threads on a single machine; Matrix raises the ceiling to 1,500. On 13 H100 GPUs the result was a 15.4× gain in tokens per second at the same reward level. On top of that, message offloading cut the network peak by about 20%.

Why it works

Removing the central scheduling point raises scalability by eliminating the bottleneck. Splitting inference out into an external service and moving to gRPC cuts network overhead and latency. Row-level scheduling raises utilization and makes token output steadier.

Who this is for

Teams working on synthetic data generation, tool-use scenarios, dialogues and other cases that need agentic scenarios to scale. Reinforcement learning and active learning researchers need generative pipelines that scale as well. The framework is also useful in production workloads heavy on tools and containers, such as support and analytics.

What to watch for

Matrix is aimed at cluster deployment with Ray, SLURM and Ray Serve. Its advantages show up in scenarios with tens of thousands of parallel tasks, where conventional solutions hit their scaling limits. The code is well organized, but it takes some understanding of how to configure the external inference services. The Matrix architecture looks solid enough to extend to multimodal agents. The authors clearly left headroom in it.

The bottom line

Matrix makes multi-agent scenarios look more like a network protocol than a job scheduler, delivering gains of 2–15× with no loss of quality in real pipelines, from multi-agent agreement to shared tool use. It is a step toward data generation that is distributed by nature, with agents as networked organisms able to evolve without a central brain.

💾 Code