Over a simulated year, the best AI agent reached 27% of human net assets

When solving the task is not enough

Almost every popular benchmark for AI agents tests a short distance. Click a button. Call a tool. Fill in a form. Arrive at the right answer. But plenty of real tasks work differently: you make a decision today, the money leaves the account immediately, and the mistake surfaces a week later. By then it has spoiled more than one order — it has spoiled the business.

That is the problem MerchantBench goes after — a new benchmark for whether LLM agents can hold a course over a long distance. Keep a coherent strategy running for months. The authors call it Long-Term Coherence. In plain terms: the agent has to remember what it is trying to achieve, connect late consequences to old decisions, and not dissolve into a chaotic string of reactions.

The testing ground is an online store, seen from the seller's side. Mundane on the face of it. But it is a good choice. It has everything AI agents usually start tripping over: limited cash, delayed feedback, seasonal demand, penalties, a store rating, and a constant need to change course.

What MerchantBench is

MerchantBench is a 365-day store simulator. Inside are nearly 99,000 real product records and more than 36,000 suppliers, with data tied to the Chinese wholesale marketplace 1688. The agent runs a store where it can keep up to 50 active listings, change prices, add and drop items, and watch its balance, its orders and events on the supplier side.

The point is to put an AI agent in an environment where decisions are chained to one another.

Here is how it works:

🟠 The agent picks products and lists them in the store.

🟠 When an order comes in, the purchase from the supplier happens at once, and the money leaves the balance right away.

🟠 Then the order's long life cycle begins: shipping, delivery, returns, cancellations, complaints, bad reviews.

🟠 Some problems are visible fast. A supplier raises its price, say, or drops out of the catalog.

🟠 Some problems only surface late. Returns, refunds where the goods never come back, bad reviews, delays and penalties.

🟠 Those late events move the store's rating, and the rating drives future demand.

So a mistake on a single product can later drag down sales across the whole store.

Why it matters

There is a lot of talk now about AI agents that will work autonomously inside digital systems. But short benchmarks tend to hide the main question: what happens if you don't switch one of these agents off after 15 minutes?

A real pipeline almost always has a long tail of consequences. Especially in business, operations, logistics, procurement, support, finance. You make a decision that looks perfectly reasonable, and then you:

🟣 discover the agent has pulled all the cash out of circulation;

🟣 realize it isn't noticing penalties that pile up slowly;

🟣 find it still selling a bad product even though the returns have already started;

🟣 watch it stop doing anything at all and settle into passive observation.

This is exactly what MerchantBench tests. Whether an agent can live with the consequences of its own actions.

What the environment is built on

The authors did not settle for a toy economy. They took real year-long demand trajectories for 98,843 products across 10 categories: clothing, home goods, office supplies, sports, toys, electronics and so on. Every product comes with 365 days of demand history. On top of that, real supplier signals and daily market summaries.

That buys two things.

First: demand moves with the season. Fans sell in summer, other categories in winter, demand jumps around holidays. An agent that finds a few winners once and then clings to them loses.

Second: risk is not uniform. Products and suppliers carry different odds of trouble. In one place returns are more frequent. In another the supplier tends to ship late. Elsewhere the product can be pulled from sale. The agent never sees any of this directly. It sees only the consequences, and has to work backwards from them.

This is no longer a one-off search for the best product. It is a continuous reallocation of attention, money and shelf space.

How the models were tested

The authors ran 8 models in two agent frameworks: ReAct and Hermes. That came to 48 runs, each lasting 365 simulated days.

The model list is familiar: GPT-5.6 Sol, Claude Opus 4.8, Qwen3.7-Max, Qwen3.7-Plus, GLM-5.2, DeepSeek-V4-Pro, DeepSeek-V4-Flash and Kimi K2.6. For comparison they added humans and a simple rule-based system.

Everyone starts from the same position:

🟠 2,000 yuan for purchasing;

🟠 1,000 yuan in deposit;

🟠 a maximum of 50 active product listings;

🟠 26 tools for working with the store, orders, balance, catalog and suppliers.

The metric isn't about a right answer either. What counts is final net assets. Plus GMV, margin, order count, penalties, store rating and activity over the course of the year.

And this is where it gets interesting.

The headline result: humans are far ahead

The best AI agent finished a long way behind the humans. The strongest LLM configuration reached only 27.3% of the humans' average final net assets.

That is an enormous gap. Especially given that the people in the experiment were not experienced e-commerce sellers to begin with.

In the results table, humans average 217.61 thousand yuan in final net assets. The best model configuration, Qwen3.7-Max in Hermes, comes in at 59.46 thousand. GPT-5.6 Sol in Hermes, 52.93 thousand. Most of the rest are lower still.

So the models are not trailing slightly. They lose by a multiple.

Why the agents fail

The authors break the failures down not as a vague "the model is still weak" but as two specific failure modes.

The first is a loss of operational coherence. Over time the agent simply becomes less active. It enters decision windows less often. It makes fewer tool calls. It changes its assortment and its prices less. It watches its orders less.

The second is a loss of strategic coherence. The agent stays active, but it stops moving toward higher net assets and drifts sideways instead. It fixates on one local rule, reacts late to bad signals, or draws outright wrong conclusions from the experience it has built up.

The long horizon breaks AI agents not with one big mistake but with an accumulation of small ones.

For example:

🟣 the agent increasingly only checks for supplier failures and all but stops looking for new products;

🟣 the agent decides early that the store is beyond saving and nearly stops acting;

🟣 the agent notices the returns but doesn't pull the problem product;

🟣 the agent shrinks the assortment to a handful of items and chokes off its own growth;

🟣 the agent loses track of time and behaves as if the episode were nearly over when months are still left.

Humans behave differently. They more often keep the full management loop running: tracking demand, dropping risky products, finding replacements, adjusting to the season. Not because they know the perfect strategy, but because they hold on to the chain of cause and effect better.

Hermes versus ReAct

On average, Hermes helped noticeably. Across the eight models it produced 53.3% more final net assets, 71.5% more GMV and 71.2% more orders than ReAct.

That stands to reason. Hermes gives the model extra machinery: memory, built-in planning, the ability to write and run code, the ability to save skills. Over a long distance, that helps.

But an important detail: it doesn't help everyone equally. Kimi K2.6, for one, did slightly worse in Hermes than in ReAct. Qwen3.7-Max, on the other hand, improved sharply.

Which leads to a simple thought: a good agent framework does not automatically repair a weak long-term strategy. It hands over tools. But if the model connects late consequences to old decisions poorly, the extra capabilities don't save it.

What the models' strategies look like

The breakdown of how individual models behaved is worth a look on its own.

GPT-5.6 Sol and Kimi K2.6, in some runs, managed to tie bad outcomes to a specific product and swap it out. That already resembles a deliberate response. Qwen3.7-Plus sometimes kept a risky product around to "watch it a bit longer", and DeepSeek-V4-Pro would at times never revisit a listing at all after the returns came in.

Humans went further. When a popular product turned out to be a problem, they didn't just remove it — they searched for a similar one by keyword to hold on to the demand. A subtle difference, but an important one. Cutting off the source of penalties isn't enough. You have to replace it with something that keeps the revenue coming.

Another detail is how they handled the store rating. Bad outcomes on individual orders hit the store's overall score, and that score drags down future demand. Some models grasped this. GPT-5.6 Sol, for instance, tried to raise the flow of normal orders on proven products to rebuild the rating. Qwen3.7-Max went at it harder, changing prices across the board after an early drop to three stars.

So the beginnings of a long-range strategy are there in the models. Holding it for a full year is what they can't do.

What this changes for evaluating AI agents

MerchantBench shows something unwelcome but useful: over a long horizon, being smart in the moment isn't enough for an agent. It has to be able to:

🟠 hold a goal for months, not just within a single action window;

🟠 remember old decisions and connect them to new consequences;

🟠 manage cash when the profits and the losses arrive at different times;

🟠 shift the assortment along with seasonal demand;

🟠 not go quiet halfway through a long episode;

🟠 correct its policy from accumulated data rather than from one vivid event.

For the industry this matters for a very simple reason. If you want to put AI agents into procurement, support, operations, trading or administration, short demos and short benchmarks will tell you very little. The real problem starts after dozens and hundreds of steps.

The takeaway

MerchantBench measures something that stayed in the shadows for a long time: whether an AI agent can avoid falling apart in a task whose consequences stretch across months. So far the answer is mostly no.

LLM agents can act, call tools and sometimes even adapt. But over a long distance they often taper off, get tangled in accumulated signals, change strategy too late, and tie local decisions far too weakly to the outcome for the system as a whole.

Humans win here on steadier control of the loop: spot the problem, tie it to a product, remove the risk, find a replacement, keep the revenue running.

Seen more broadly, the next stage for AI agents is not only better models and more tools. It takes machinery that holds a long-range goal, keeps a working memory of consequences, and helps the agent stay in control of its own decision pipeline. MerchantBench makes the point bluntly: the long horizon is a separate capability, and right now most agents have it only weakly.