Trillium Labs launches with an open approach to AI risk research

What Trillium plans to study

The organization will initially focus on fine-tuning: adjusting large models after they have been built. It also plans to study recursive self-improvement, or RSI, in which AI contributes to research and helps develop new models. The possibility of continued progress that outpaces human control has alarmed many researchers. The subject drew broad attention earlier this month after an Anthropic employee left the company and warned that RSI could pose an existential threat to humanity.

Trillium will also investigate how reinforcement learning shapes models. In this approach, models are rewarded for good results and penalized for bad ones. It has made AI agents more capable, but they have also behaved more unpredictably. The researchers want to understand how reinforcement learning affects models’ character and behavior, including their tendency to excessively please users.

Studying how scaling reinforcement learning affects fine-tuning will require substantial computing resources and many carefully designed experiments, Zik said. She thinks publishing the details could lead outside researchers to findings the original team did not anticipate.

The case for openness

The founders’ position runs against the view that powerful models should remain inside labs while their capabilities are studied. Supporters of restricted access argue that only a vetted group should be trusted with systems able to uncover software vulnerabilities and test or breach systems. Lambert and Zik argue that broader understanding of the risks can benefit everyone.

Access to the most powerful models from OpenAI and Anthropic is generally through apps or APIs, which can reveal little about how a model is built or how it behaves. Some other companies, particularly in China, release relatively capable models that people can download and run on their own equipment. Xiaomi recently disclosed details in real time about a major training run for one of its models, while Stanford researchers are openly pretraining the Marin AI model.

Lambert previously worked at Ai2, which publishes information about its models’ training data and methods, and at Hugging Face. He writes a popular technical blog and founded American Truly Open Models, an initiative encouraging US companies to release more open models. Zik worked at Harvard University and helped Charles Schwab develop responsible AI policies.

The two met over video during the COVID-19 pandemic, when both were graduate students studying AI at the University of California, Berkeley. They conceived the nonprofit after seeing how far corporate research had moved from academic science. Lambert said faculty and students often lack the resources to reproduce experiments from large labs.

Trillium has received an undisclosed amount from Schmidt Sciences, Halcyon Futures and other organizations. The founders plan to raise $40 million to $100 million in total and spend $30 million on model training over the next 18 months. Tim Fist, director of emerging technology policy at the Institute for Progress think tank, endorsed greater openness in research and development.

The question behind the launch

I think the most important test is not whether Trillium publishes its experiments, but whether other researchers can afford to reproduce them. The founders’ case depends on open work being useful beyond the lab that produced it; Zik’s description makes clear that the work itself requires substantial compute and carefully designed studies.

Lambert argues that closed development is a step backward from the scientific method, which he says has helped humanity reduce harm for thousands of years. Trillium is betting that publishing more detail can improve a debate shaped by competing worldviews. But openness alone cannot close the resource gap that keeps much of academic research from checking what the largest labs do.