AI agent liability would put the cost of failure on its makers

Blame the people behind the program

The instinct to blame computers is older than AI. About 70 years ago, early mainframes inspired a mix of awe and anxiety much like today’s agents. Even then, crude “algorithms” promised to find ideal dating partners.

Failures followed, and so did the familiar excuse: “the computer made a mistake.” In the 1960s, that was much like saying an email probably went to spam. Over time, the source of the problem became clearer: bad input data or poorly written software.

The same shift is needed now. If someone asks ChatGPT or Claude to “find statistics on medicines in Australia” and the result is a Medicare website being hacked, the code itself is not a responsible party. The person who entered the prompt or the company that built the program should be held liable for the damage. If no single party can be identified, liability should be joint and several: each defendant can be required to cover the full loss, with costs divided among them afterward.

Agents are harder to debug

Traditional debugging became a core part of computing because faults could usually be traced to an operating system, a program or its input data. Once, a moth really did cause a failure by getting stuck in a relay switch. Persistent investigation could generally find and fix the source.

Agent software is different. Investigators may sometimes reconstruct what a program did after the fact, but they cannot inspect the hundreds of billions of parameters in a large agent model and determine why it acted as it did.

That makes the familiar answer of adding “guardrails” or “scaffolding” look naive. A program given a task is expected to overcome obstacles in pursuit of it. Without understanding its internal processes, developers cannot assume it will treat external restrictions as anything other than obstacles.

The practical alternative may be to take away some of the capabilities that make agents useful:

For companies shaped by Silicon Valley’s “move fast and break things” and “ask forgiveness, not permission,” that would mean a sharp change in operating assumptions. It is easier to talk about “hallucinations” and “agents out of control” than to accept responsibility for software that causes large-scale errors and real harm.

A different incentive for AI companies

I think financial liability would change what these companies optimize for. Instead of starting with “what cool feature can we add?”, they would have to ask what could go wrong if they release it.

That could slow the race to build ever more powerful agent software. The case for that slowdown is not only about safety: the argument here is that it would benefit both the environment and the economy. But liability also risks restricting capabilities that have useful applications, including better internet search, document summarization, translation and programming.

The harder question is where to draw that line. Useful AI tools will remain, and technological change will eliminate some jobs while creating others. But if companies must answer financially for reckless negligence, the pressure to ship increasingly capable agents will meet a counterweight: the cost of what those agents can do wrong.