How AI Predicts Brain Activity

How to Teach a Model to Predict Brain Responses

As someone watches a TV show, their brain processes images, speech, music, and the meaning of what’s happening. Can we use the video, audio, and subtitles alone to predict how activity in different brain regions will change from moment to moment?

Models like this already exist. But they’re usually trained cautiously: researchers take a large neural network, freeze its weights, and train only a small module that maps the model’s internal representations to brain signals. A new approach lets brain data make small adjustments to the network itself—and shows that there’s no need to retrain the whole thing.

In this study, researchers used lightweight fine-tuning on multimodal models that understand video, audio, and text. On brain MRI data collected while people watched movies, the approach improved predictions across nine models. A specialized computing method made training faster and enabled the researchers to build ensembles of hundreds of models.

Why Change a Large Model?

A brain-encoding model takes a stimulus—such as a scene from a movie—and predicts the brain’s response. To do that, it needs good representations of what’s happening on screen and in the audio. Large pretrained models already extract these kinds of features, so researchers often freeze them and train only an output layer.

This conserves data. Brain recordings are expensive, and there are usually too few examples to reliably update billions of parameters. Full fine-tuning can waste the available data and even hurt performance. But a frozen model has a drawback: its representations aren’t specifically adapted to predicting brain activity.

The authors propose a middle ground. The large model’s weights stay fixed, while small add-on modules learn to adapt how it works. One such method is called LoRA. It makes a limited change to the model using a small number of parameters. Brain data can therefore shape the model’s representations without requiring the entire model to be retrained.

A model can stay frozen or be adapted with small modules; the approach scales to larger models and many specialized variants.

How to Cut Computation by 350×

The experiments used brain-activity recordings from four people as they watched episodes of Friends. The data was split into a training set, a validation set covering season six, and a test set covering season seven. The researchers also assessed how well the approach transferred to different kinds of movies. Brain activity was represented as signals from 1,000 regions.

The main computational challenge was the length of the videos. To predict a brain signal at a particular moment, the previous approach repeatedly processed overlapping clips from the movie. The same context went through the model hundreds of times. The authors replaced this with a single pass over a time window: the model takes in the whole clip and outputs features for many moments at once.

This speedup is possible because of how the model works. It processes the sequence using a causal mask, so the representation for any given moment can’t use words or frames from the future. That means computations can be reused across neighboring moments, rather than starting over from scratch for each window.

On a single GPU, feature extraction fell from about 700 hours to 2. That sped up video and audio processing specifically: training the full system took about 8.5 hours, compared with roughly 702 hours using the previous method. But accuracy also matters. A single pass without additional context wasn’t enough, especially at the start of a window. The authors added up to a minute of material before the target clip, giving the model the context it needed while preserving accuracy.

A single pass over a time window replaces repeated processing of overlapping clips.

Lightweight Fine-Tuning Improves Predictions

In the main experiment, the researchers added LoRA modules to Qwen3-Omni and trained them together with a compact output layer. In total, they trained 61.2 million parameters. The output module in the previous leading approach—which kept the large model frozen—contained 1.1 billion parameters.

On the test set of unseen episodes, the new model achieved a correlation of 0.327 between its predictions and measured brain activity. The previous result was 0.310. On new movies the model hadn’t seen during training, the scores were 0.228 and 0.217, respectively. Correlation measures how closely changes in the predicted signal track changes in the measured signal.

The improvement wasn’t limited to one model. The researchers tested nine multimodal models ranging in size from 4.7 billion to 104 billion parameters. Lightweight fine-tuning improved every one of them. Model size alone, however, didn’t explain performance: more parameters didn’t always lead to more accurate predictions.

The authors also compared LoRA with full fine-tuning. Updating all 20.3 billion parameters brought no improvement, despite requiring 330 times more parameters to be trained. With the data available, a small adjustment was enough.

Add-on modules improved performance across all nine models tested.

The researchers also tested how well the approach transferred to new participants. A model trained on data from three people retained almost all of its improvement on a fourth person, whose data it hadn’t seen during fine-tuning. They also tested it on a separate dataset featuring different participants, a different scanner, and a different movie. Without any additional training, the new module increased the proportion of explained brain signal by 11.1 percentage points relative to the ceiling set by measurement noise.

What the Model Learns About the Brain

A multimodal model combines information from images, audio, and text. That makes it hard to tell which input influenced the predicted activity in a particular brain region. To measure each stream’s contribution, the authors removed the video, audio, or text in turn and checked how prediction accuracy changed.

The results matched what neuroscientists would expect. The visual cortex responded mainly to video, while the auditory cortex responded mainly to audio. In the lateral temporal regions, audio also mattered more than text. So even within a single multimodal model, it’s possible to identify links between types of stimuli and brain regions.

After fine-tuning, the contribution of audio to predictions rose by an average of 39%, while the contribution of text fell by 17%. The model’s internal workings changed, too: information from one stream more often appeared in representations associated with another. For example, video content could influence features in positions occupied by audio tokens.

This doesn’t mean the model reproduces how the brain works. The authors show a narrower result: lightweight fine-tuning changes how the model uses different kinds of input, and those changes are linked to which signals help predict activity in particular brain regions.

Contribution analysis links video, audio, and text to different brain regions and shows how fine-tuning changes how they interact.

Why Train Hundreds of Models?

Even an accurate model can fail where another succeeds. So the authors trained 261 variants with different settings, output layers, and base models, then combined their predictions.

To avoid running hundreds of separate training jobs, the researchers developed a computing kernel that could work with multiple LoRA configurations at once. Using the same shared base model, it updates many adapters in a single pass. With 32 adapters, a training step was almost 30 times faster than training them sequentially. It took roughly as long as training one adapter on the same combined set of examples.

The 261-model ensemble achieved a correlation of 0.343 on unseen episodes and 0.248 on new movies. The authors report that, when the paper was being prepared, this was the best result on the public Algonauts competition leaderboard. But the ensemble’s accuracy depended not just on the number of models, but also on how good they were individually and how differently they made mistakes. New output layers and base models added more diversity than changing the random initialization.

Training many adapters together speeds up ensemble building; accuracy depends on the number of models, their quality, and the diversity of their errors.

Conclusion

A large model can be adapted to limited brain data without retraining it in full. In these experiments, lightweight fine-tuning improved predictions across nine multimodal models and outperformed the previous approach while training far fewer parameters. Single-pass video processing reduced computational costs, and joint adapter training made it possible to build a large ensemble.

The results are still limited: the main evaluation used movies and data from four participants. Transfer to a separate dataset is encouraging, but it doesn’t show how the method will perform with other types of stimuli or brain recordings. The study offers a practical path forward: start with a strong pretrained model, adapt it using small modules, and combine different variants when prediction quality matters most.