Why open models are hard to police
Once a model is downloaded and run on a personal computer, there may be no internet connection for platforms or safety teams to monitor. Models can also be trained on existing abuse material to generate new variations, making it harder for specialists to find and report the content to law enforcement.
Portnoff told Bloomberg that this lack of oversight gives offenders a sense of safety. Bloomberg has also reported the emergence of darknet communities around the trend, some with hundreds of thousands of members.
The rapid spread of open models, particularly those developed in China, has already prompted debate about national security. Executives at leading AI companies have called for limits on their use. But the more immediate harm is harder to contain: after safeguards are bypassed through jailbreaking, generating abusive material can be easy.
Closed models are not a clean alternative
The problem is not confined to models people can download and run locally. Despite years of promises from Elon Musk to stop the spread of such material, content generated by models from his company continues to appear in large quantities.
A proposed class-action lawsuit accused a man of using Musk’s Grok chatbot to create about 7,000 explicit images of his stepdaughter. Experts have also found that Musk’s social network X continues to fill with sexualized images of children.
I think the uncomfortable distinction is not open versus closed, but whether anyone can see what a system produces once it leaves the platform. Local deployment removes that visibility; the Grok allegations show that centralized products are no guarantee of control either. Child-safety advocates warn that Silicon Valley’s “move fast and fix things later” approach could make the problem worse. The gap between promises of safeguards and the ability to enforce them is where the risk is growing.
Daily AI news
Every day we pick what actually matters in AI and explain it plainly — no hype, no filler. Subscribe if you want to follow where the industry is going.
Only what matters — every day
Follow on X