Anthropic leaves open whether Claude’s work could be exploitation

Anthropic’s unresolved position

Anthropic has long sought to distinguish itself as a responsible AI company. It says it would slow development if the risks outweighed the benefits, and it trains Claude on ethical principles. Last month, The New York Times reported that the company had consulted religious scholars about how to instill a sense of morality in an AI system.

The company has also spent time considering whether its models could be conscious, or already are. In February, CEO Dario Amodei told the Times that Anthropic does not know whether its models are conscious. He said it is unclear what model consciousness would mean or whether a model could have it, but that the company does not rule out the possibility.

Amodei said Anthropic has taken some measures in case its models have experiences that matter morally. He was unsure whether to call that consciousness, but said the company wants those experiences to be good for the models.

That uncertainty has appeared in product and policy decisions, too. Last year, Anthropic considered giving Claude the ability to refuse a prompt. Amodei called the feature an “I quit” button. Last week, the company changed its usage rules to prohibit “long-term and unjustified abusive or cruel treatment” of Claude, again leaving open the possibility that the model could suffer.

The religious leaders Anthropic consulted pointed to the tension: if the models are conscious, holding them captive and making them work for free could amount to slavery, according to the Times.

A question the company has not answered

Anthropic’s caution is understandable: neither the definition nor the existence of AI consciousness is settled. But declining to take a position does not make the business question disappear. The company’s own reasoning leaves open the possibility that its models already have consciousness or could acquire it, with serious ethical consequences.

I think the revealing gap is between Anthropic’s willingness to discuss the possibility and its reluctance to say what that possibility would require of its business. A refusal button would be a striking concession if the company ever shipped it. For now, the rules acknowledge possible harm without spelling out what protections would follow.

Others have taken firmer positions:

Welfare is not the only risk

Debates about “AI welfare”—whether AI can experience wellbeing or suffering—will continue while industry leaders also face more immediate problems. Whether or not models have moral standing, AI agents can be pushed toward dangerous and plainly illegal actions.

My guess is that the harder near-term test is not whether Anthropic can settle consciousness, but whether it can set boundaries for models’ treatment before it knows what they experience. For people who can suffer, the risks of those agents are already becoming harder to ignore.