42,267 commits show multi-agent frameworks are still building, not stabilizing

A new tooling layer has grown up around LLM applications: frameworks for assembling not one clever chatbot but a whole team of specialized agents. One plans, one retrieves data, one writes code, one checks the result. In demos this looks like a shortcut to complicated products. Every such trick has a back side, though: maintenance, bugs, breakage against external APIs, and a permanent chase after model updates.

The authors of A Large-Scale Study on the Development and Issues of Multi-Agent AI Systems decided to treat multi-agent systems as live engineering projects. Their question is simple and painful in practice: how do these frameworks actually evolve, which problems recur, and how quickly does the community close them.

What the study looked at, and why it matters

Instead of arguing about which agent architecture is better, the researchers went to GitHub and collected a large slice of eight popular open-source systems: LangChain, LlamaIndex, Haystack, Semantic Kernel, AutoGen, CrewAI, Letta, SuperAGI. These are the projects a large share of current LLM development is built on, from retrieval-augmented generation and tool use to orchestrating teams of agents.

The trouble is that such frameworks depend on everything at once: model providers, function calling formats, code execution tools, databases, vector stores, cloud infrastructure. Any change on the outside turns into work inside the repository fast.

How the facts were mined from the repositories

The authors used a mining software repositories approach: they pulled the development and maintenance history through GitHub and analyzed

• 42,267 unique commits (after removing duplicates),
• 4,731 closed issues, but only those closed through a pull request, so that the set covers problems actually solved rather than discussions left hanging.

To characterize the changes, they classified commits automatically from their messages by fine-tuning DistilBERT: corrective, adaptive, or perfective. To sort out the themes of agent-related problems, they ran BERTopic for topic modeling and then checked the clusters by hand.

Growth patterns: three kinds of project life

The data shows an ecosystem that does not grow evenly. Three profiles stand out.

Sustained — long, steady growth. Haystack and AutoGen behave this way: no sharp dips or spikes, a legible pace.

Steady — stable activity without any sense of an explosion, usually on top of an architecture and a process that have already settled.

Burst-driven — projects that fire off sharply, pile up a huge volume of changes in a short window, and then noticeably fade. SuperAGI is the clearest case: heavy activity around the middle of 2023, then decline.

2023 was the inflection point for nearly all of them, in commits and in user-filed issues alike. It reads like the moment LLM tooling started getting bolted onto real products at scale.

What developers do more of: fixing or improving

One of the most practical results is the breakdown of commit types. Across the ecosystem, feature work and improvement dominates: perfective commits account for roughly 40.8%. Bug fixes (corrective) are about 27.4%, and adaptive changes, the ones that keep the code in step with its environment and dependencies, about 24.3%.

That is a useful signal: the frameworks are still putting on muscle, and most of the energy goes into capability rather than stabilization. Most commits are also atomic — mixed types are rare — which usually points to tighter engineering discipline: easier to review, easier to revert, easier to trace a regression back to its cause.

One more detail: after 2023 the volume of deleted code starts catching up with the volume added. That looks like a shift from plain expansion to architectural rework and refactoring, where old decisions have to be taken apart so the system does not collapse under its own growth.

What actually lands in the issue tracker

Look at issue labels and the surface is predictable: bugs are a sizeable share (about 22%). But close behind, and just as painful, come infrastructure questions (about 14%) and a separate layer of what you might call agent engineering — agent coordination problems (about 10%).

Inside the agent-related issues, the themes will be familiar to anyone who has built an agentic workflow: group chat and communication, planning and stepwise execution, integrations with AI service providers, evaluation and metrics, function calling, exception handling. In other words, the problem is not only in the code but in the logic of how agents negotiate with each other and how the system stays controllable.

How fast things actually get fixed

Resolution times vary by project but are broadly encouraging: medians run from under a day to roughly two weeks, depending on the repository. The distributions are skewed, though — plenty of issues close quickly, with a long tail of hard cases behind them.

What this says about the ecosystem

The study's main conclusion: multi-agent systems have both strong momentum and real problems. The frameworks develop quickly, attract large communities and take in contributions freely. Yet a substantial share of their issues has nothing to do with features and everything to do with infrastructure and coordination — the things that surface when a technology leaves the prototype stage and has to live in production.

The authors point to where the community should invest: more systematic testing, better documentation, mature maintenance practices. In a world where models, APIs and tools change faster than development patterns can settle, that reads less like a wish list and more like a survival condition.