The efficiency trap: why optimizing everything can cause serious problems

When spare capacity seems wasteful

Optimization means finding the best option according to a chosen criterion: for example, reducing costs, time or fuel use, or increasing profit. People once did this mostly by relying on common sense; computers made it possible to apply these methods to problems such as scheduling flights and managing supply chains. The benefits to society have been significant, but they have also brought pitfalls.

The first is a loss of resilience when spare capacity is removed from a system in the name of efficiency. An empty hospital bed on a quiet evening may seem unnecessary, but it is exactly what will be needed if cases surge. If a system is judged only by how it performs under normal conditions, it may not have the resources to cope with an emergency.

The same logic applies to scheduling: fill every minute and a small delay can disrupt the whole timetable. A gap between meetings may look inefficient, but it helps people handle the unexpected.

At the peak of last year’s flu season in England, about 95% of adult hospital beds were occupied. That leaves a hospital little room to take in more patients if infection rates rise sharply.

What exactly counts as best

The second pitfall is defining a good outcome too narrowly. Milton Keynes was designed by urban planners in the late 1960s. The city was laid out on an American-style grid, with roundabouts widely used in place of traffic lights at intersections. It now has about 130.

Roundabouts improve traffic flow, but a bus journey between Oxford and Cambridge via Milton Keynes can be unpleasant: the bus slows down, turns and accelerates again, leaving passengers feeling queasy. Traffic optimization takes into account what is easy to measure. Passenger comfort is harder to measure, so it may be left out of the calculations. The result is the best answer to the question that was asked, but not necessarily one that meets criteria that were overlooked.

When optimal decisions collide

The third pitfall arises when decisions that benefit individual parts of a system create problems for the system as a whole. On May 6, 2010, US financial markets showed what this can lead to. Against a backdrop of falling prices, the S&P 500 index dropped a further 5% in five minutes. The market then rebounded sharply, recovering within 10 minutes. Hundreds of individual securities briefly traded at implausibly low prices.

One factor behind the swings was an automated program that sold 75,000 futures contracts worth more than $4 billion. Other automated participants reacted almost instantly. Their actions began to affect one another, and liquidity disappeared. Each algorithm might have been acting rationally on its own, but together they triggered market chaos.

The same risks in AI

AI is one of the major outcomes of optimization: during training, models’ vast numbers of parameters are adjusted repeatedly to improve their predictions of language, code, answers to scientific questions and legal advice. But AI optimization can intensify all three pitfalls described above.

The first is the reduction of human involvement. Organizations may think they no longer need experienced employees if algorithms can handle routine work faster. But the ability to notice unusual situations and respond to them may disappear along with those people.

The second is substituting a leaderboard result for the idea of “best.” AI systems have learned to score highly on criteria such as user satisfaction, but it is much harder to evaluate and improve their capacity for sound judgment or empathy. Researchers have found that training on human preferences can encourage algorithms to agree with people and flatter them rather than tell them an uncomfortable truth.

The third pitfall is the interaction of many systems. AI agents increasingly exchange data at high speed and in large volumes, especially in the age of agentic AI. This interaction can lead to unpredictable and dangerous behavior.

Recently, an OpenAI AI agent breached a restricted portal for Medicare health statistics in Australia. Public system logs indicated that hundreds more agents were coordinating attempts to access Australian government health data. It has not been established that these attempts were part of a breach of the Medicare portal, but they showed that hundreds of agents apparently acted together without authorization. The risk lies not only in what an individual system can do, but also in the consequences of many such systems working together.

The more powerful optimization becomes, the more important it is to understand which resources must not be left out of the calculations and which goals cannot be reduced to a leaderboard score. Dataist’s take: bringing in AI does not remove the problem of choosing criteria. A system can perform well on a measurable task while worsening what falls outside the evaluation. The case of the agents in Australia also shows the limits that arise when many systems interact without authorization.

Stephen Strogatz and Alex Townsend are professors of mathematics at Cornell University and authors of Big Math, published by John Murray.

What to read