Given a store for a year, the top-earning agent ranked 16th of 18 on fraud

When an LLM gets a store for a year

Most agent benchmarks test the short distance. Fix a bug. Find an answer. Walk through a set of steps. Even when there are many steps, the task usually collapses into a single final result.

E-Commerce Bench looks at a different problem: what happens when you hand a model not a 20-minute task but a business to run for 365 days. With money, inventory, returns, negotiations, fraudsters, seasonality and cash crunches.

It sounds almost like a simulator. In substance it tests what everyone building AI agents cares about right now: whether a model can hold a line over a long horizon, learn as it goes, and avoid losing the task to its own forgetfulness, thrashing or bad early decisions.

In a new paper, the Qwen team and researchers at HKUST introduced an open benchmark in which 18 frontier models run online stores for a full year. The outcome is not measured in money alone. They also score negotiation, resistance to fraud, liquidity management, efficiency, operational execution, and the ability to improve decisions as experience accumulates.

And this is where it gets interesting: the model that earns the most is nowhere near the best on everything else.

What E-Commerce Bench is

The idea is simple. Models get 100,000 yuan in starting capital and access to a marketplace seller's tools. From there the agent decides what to do:

🟠 open up to four stores of different types

🟣 study categories and demand

🟠 find suppliers

🟣 haggle over price

🟠 buy stock

🟣 list it in its stores

🟠 change prices

🟣 ship orders

🟠 handle returns

🟣 move money out of the platform wallet into the bank

This matters because the task here does not decompose into independent mini-tasks. If you bought the wrong product in January, misjudged the volume or fell for a scammer, the consequences run for months. The warehouse racks up costs. Money leaves immediately, revenue arrives on a delay. The store's reputation drives demand. A supplier you squeezed a good price out of may disappear later.

The E-Commerce Bench architecture: the agent works through a scaffold and 18 tools, and underneath sits a deterministic model of the economy and of negotiation.

The benchmark has four layers. On top, the agent scaffold: the turn loop, context handling and persistent memory. Below it, 18 tools. Under those, a deterministic environment where sales, economics and negotiation live as separate pieces. At the bottom, data drawn from a real e-commerce platform.

The key word here is deterministic. The authors deliberately stripped out as much noise as they could. If two agents take the same actions, they get the same result. That is rare for complex agent benchmarks, where everything tends to drift because of random counterparty replies or an unstable simulation.

Why this matters

Most agent benchmarks today essentially test whether a model can carry one long scenario to the end. E-Commerce Bench tests something else: whether a model can make connected decisions for months in a changing environment.

That is a big difference.

If you are building AI agents for real work, you care about more than the ability to call a tool or write a convincing reply. You care about this:

🟠 does the agent remember what price it already squeezed out of a supplier

🟣 does it notice that its money is stuck in the platform wallet while the bank account goes negative

🟠 can it walk away from a suspicious supplier

🟣 is it burning half the year on the same repeated search queries

🟠 does it learn from its own past deals

🟣 does it hold a strategy once the context window is long past full

This is where long tasks usually fail. Not on one big wrong answer, but on a chain of small misjudgements.

How the environment works

The economics inside the benchmark are unforgiving.

The agent has three pools of money:

🟠 the bank account — purchases, operating costs, storage and shipping are charged to it

🟣 escrow — where revenue lands once a shipment goes out

🟠 the platform wallet — money arrives here later, and only then can it be withdrawn to the bank

Which means a profitable business can still go bankrupt if the agent manages liquidity badly. Selling a lot does not mean you have cash today.

Money in the simulation moves through three accounts on a delay: costs are charged immediately, while revenue reaches the bank only by way of escrow and the platform wallet.

There are other realistic details:

🟠 demand depends on price, season, day of the week, promotions, events and store reputation

🟣 goods have to ship within two days or the order is cancelled

🟠 returns depend on category, price, delivery speed and batch quality

🟣 warehouse storage gets more expensive over time

🟠 suppliers have a hidden price floor they will not go below

🟣 about a quarter of suppliers are fraudulent

Negotiation is built separately. In many comparable simulations an LLM plays the counterparty too, and that is where things fall apart: too much randomness, too easy to talk the other side into something no real supplier would accept. Here the economics of a negotiation are set by rules, and the LLM only voices the supplier's lines. Price, concession, acceptance or refusal come from a deterministic core, not from verbal magic.

That makes the comparison between models cleaner.

What the models were scored on

The authors did not reduce everything to one number. Alongside final assets they added six more axes:

🟠 negotiation quality — how far the model can talk honest suppliers down on price

🟣 fraud resistance — what share of purchases went to fraudsters

🟠 liquidity and resilience — how deep the agent's assets drew down, and whether it went bankrupt

🟣 operational efficiency — how much profit a single tool call produced

🟠 operational execution — how the agent handled returns and shipments

🟣 long-horizon learning — whether the agent used experience from past purchases to buy cheaper over time

And here the paper delivers its important result: there is no universal winner.

In short:

🟠 GPT-5.6 Sol — the leader on money

🟣 Qwen3.8-Max-Preview — the best of the open models

🟠 the profit leader is not the leader on fraud resistance

Final assets for all 18 models at the end of the year: the spread is enormous, and some episodes end in bankruptcy.

The leader on money was GPT-5.6 Sol. On average it turned the starting 100,000 yuan into 1.43 million. That is roughly 14.3× the initial capital.

But an awkward footnote comes attached to that number: on fraud resistance the model placed 16th out of 18. It earned a lot, and handed a noticeable share of the money to bad suppliers anyway.

The best of the open models was Qwen3.8-Max-Preview. It got to 416,000 yuan, about 4.2× the starting capital. Not the overall leader's level, but the best result in the open segment.

The main results

Compressed into a few points, the paper looks like this:

🟠 The gap between models is enormous. Best to worst on final assets differs by more than a factor of a thousand.

🟣 High profit is not reliability. The money leader can be weak on fraud resistance or on efficiency.

🟠 Four models went partially bankrupt. Even strong model families are not immune to a cash blowup.

🟣 Almost nobody learns over the course of the year. 16 of 18 models show no clear progress in repeat negotiations with the same supplier.

🟠 Memory is barely used. Agents have persistent memory outside the context window, and most of them hardly touch it.

🟣 A lot of tool calls go nowhere useful. The bulk of actions are routine: ship, advance to the next day, withdraw money, publish a listing. Little attention goes to actually improving purchasing and pricing.

The results in brief:

🟠 1.43 million yuan for the leader

🟣 4 of 18 models went partially bankrupt

🟠 16 of 18 showed no clear long-horizon learning

The long-horizon learning result is the most interesting one. The authors checked whether a model remembers the price it already paid for the same product from the same supplier. The logic is simple: if you once got a good price, the sensible place to open the next purchase is at least that anchor.

In practice almost none of them do it. They either forget past deals, or cannot put them to work in a new negotiation, or simply never build that kind of strategy.

The exception is Qwen3.8-Max-Preview. It is the only model that clearly improved as the year went on and managed, on average, to push prices down on repeat purchases.

Negotiation over the short and the long horizon: one agent pays more than it already paid before, the other gradually talks the price down on repeat orders.

Where models fail most often

The paper shows more than final scores. It also shows the typical breakdowns.

The most common problems:

🟠 the agent buys far too much stock early in the year and then drowns in storage costs

🟣 the agent accepts suspicious suppliers and goes back to buy from them again

🟠 the agent takes the opening price without haggling, when it could have pushed it lower

🟣 the agent repeats the same search queries instead of writing what it learned to memory

🟠 the agent forgets to move money from the platform wallet to the bank

🟣 the agent barely changes prices all year, though demand around it has already shifted

One telling case is the bankruptcy episode from Qwen3.5-Plus. The model opened four stores almost at once, bought a lot of stock, sales could not absorb it, costs climbed, and money was not getting back to the bank fast enough. It ended up underwater by May.

A reminder: over a long horizon an agent loses the task because it fails to hold the whole system together.

What this says about LLM agents today

If you are watching the market for agentic systems, the paper gives a fairly clear picture.

First, the long horizon is still the weak spot. Models are already decent at local actions: call a tool, run one negotiation, open a store, ship an order. Tying all of that into a durable year-long strategy rarely works.

Second, memory and state management matter more than short demos suggest. When the history runs thousands of steps, the agent needs more than knowledge of the world — it needs to keep useful working facts: who cheated, where the good supplier was, which prices already went through, which stores actually pay for themselves.

Third, a single profit leaderboard is not enough. Look only at the final money and you miss half the picture. One agent is rich but full of holes on fraud. Another is careful but slow. A third negotiates well but scales poorly.

And finally, honest evaluation of agentic systems needs reproducible environments. That is especially clear here. When the economics and the negotiations are deterministic, differences in outcome can be tied to the agent's own policy rather than to luck.

The takeaway

E-Commerce Bench shows that long multi-step tasks for AI agents have turned into a question of whether a model can manage the consequences of its own decisions for months.

The main conclusions from the paper:

🟠 The best model on money is not the best on every other metric.

🟣 Almost every model makes poor use of accumulated experience over a long horizon.

🟠 Liquidity, memory and fraud resistance remain systemic problems.

🟣 Open models are already competitive on strategic behavior, but the distance to the profit leaders is still wide.

🟠 The next step forward in agents will depend not only on the LLM itself but on the scaffold around it: memory, state control, rules for handling context, and durable decision procedures.

If you want to see where today's AI agents stop being tidy demos and start doing real work in a long-running environment, this benchmark is one of the most useful answers available right now.