Taking notes, not reasoning, separates the agents that can run a startup

Agents do well on short tasks. Over a long horizon they are undone by memory, inconsistency and an inability to stick to a strategy.

AI agents have gotten decent at problems that take a dozen actions, a couple of tool calls and an answer. Stretch the task to hundreds of steps — the length of real work — and it gets interesting. Early mistakes compound, feedback arrives late, and the model's context window is finite. The agent ends up behaving as though it has forgotten its own rules.

The authors of ycbench: Benchmarking AI Agents for Long-Term Planning and Consistent Execution propose testing exactly that: the ability to hold a strategy and execute on it consistently.

A simulated startup

In ycbench the agent runs a startup through a full simulated year. There is $200k in starting capital, a team of employees with skills across domains (training, inference, research, data engineering), a market of contracts, deadlines, and a monthly payroll that keeps growing. That alone forces the agent to think ahead: a good contract feels like a win today, but two months later the payroll catches up and sinks the company.

The main trap is that roughly a third of the clients are hostile. These are not "bad vendors you are warned about in advance" but contracts that look fine and then, once accepted, balloon in scope until they are close to impossible to deliver. There is no way to look this up: the agent has to infer the pattern from a run of failures and — the hard part — remember it.

The interaction history is truncated to the last 20 turns. Anything the agent has not written into its long-term notes eventually disappears. So ycbench tests more than reasoning: it tests the loop of noticing a problem, writing it down, and using it later.

How the benchmark works

The agent works through a CLI: it looks at the task market, accepts contracts, assigns employees, advances time. The environment is partially observable — finances and events are visible, but much of the rest has to be queried explicitly. The reward is how much money is left at the end of the year, or bankruptcy if the balance goes negative.

A decision here can take weeks of simulated time to pay off. Deliberately building up trust with a single client lowers future workload and opens access to more lucrative contracts, but at first it just looks like dull routine.

How 12 models did. Spoiler: not all of them survived

The researchers ran the simulation three times across 12 models. Only three of them consistently grew their starting capital: Claude Opus 4.6, GLM-5 and GPT-5.4. The rest barely stay afloat, and some go bankrupt.

Claude Opus 4.6 finishes at $1.27M, with GLM-5 right behind at $1.21M — and GLM-5 gets nearly the same result at markedly lower inference cost, a gap the paper puts at roughly an order of magnitude.

The gap opens early, by February or March of the simulation: the strong models collapse uncertainty into a workable strategy faster. They settle on one or two reliable clients, build up trust, reduce their future workload and set off a snowball effect. The weak models spread their effort across the market, keep taking on new contracts, fall behind, and end up with payroll day working against them.

The strongest factor in success is not IQ, it's notes

The most surprising and most important finding in the paper: the use of long-term notes is the single strongest factor in success. The winner is not the model that reasons brilliantly once, but the one that keeps writing down rules and revising them as it gains experience — who to avoid, how to judge whether a job is feasible, which employees to put on which domains.

The second cause of failure is not recognizing hostile clients, which accounts for about 47% of bankruptcies. Many models accept those contracts even more often than the clients' natural share of the market (32%), because they never build an explicit client-selection policy and never commit their conclusions to memory.

What this changes in agent development

Ycbench shows that "long-horizon coherence" is a concrete pipeline: notice a signal → write it down → retrieve it later → execute consistently. On an ordinary benchmark a model can be carried by one good stretch of reasoning; winning here looks much more like managerial discipline.

The benchmark is also built to be useful: reproducible, configurable and open. And it makes it possible to compare approaches to memory and to execution honestly — not just the plan, but control over carrying it out across hundreds of steps.

In their error analysis the authors show distinct failure modes: one model reasons correctly but overloads its employees; another understands that it is close to bankruptcy and then barely acts; a third acts constantly, but on a mechanical loop, learning nothing from its failures.

The conclusion: AI can help build a startup, but not run one on its own — over a long horizon it is undone by memory, inconsistency and an inability to follow an overall strategy.