Google trains StreamVibe-VL to know when video needs an assistant

Training an assistant to hold back

To measure the problem, the team assembled a benchmark of about 4,100 examples, drawn from more than 183 hours of recordings of everyday life and software use. Gemini 2.5 Flash proposed when an assistant should help and what it should say; four evaluators had to agree on each example.

The benchmark covers three kinds of situations:

In one scenario, someone takes apart a toy excavator and asks for progress updates, then switches three minutes later and asks the assistant to check the quality of the work.

StreamVibe distinguishes between explicitly stated, unspoken, and changing user intentions during a task. | Image: Google Deepmind

Source: the-decoder.com

The researchers then trained StreamVibe-VL, which appears to be based on Alibaba’s Qwen3.5-9B. Every second, it decides whether to stay quiet or respond. A second reinforcement-learning stage teaches it to hold back rather than interrupt constantly. A separate memory system tracks unfinished requests and completed steps, but the model itself decides whether to speak and what to say.

A separate system tracks requests and task status, while the model itself decides when and what to say. | Image: Google Deepmind

Source: the-decoder.com

Long videos still expose the weakness

The researchers report that StreamVibe-VL improves response timing and quality by 10.7% over other models. The memory system is estimated to raise proactive interaction quality by a further 12.3% while reducing unnecessary interruptions.

StreamVibe-VL beats the baseline and other open-weight models, though Google’s Gemini 3.8 Flash scores higher on several tests. The authors say the specialized training does not weaken general video understanding.

As video length increases, the models are significantly less likely to time their responses correctly; the harness primarily reduces unnecessary interruptions. | Image: Google Deepmind

Source: the-decoder.com

The harder test is not deciding whether to answer at a single moment. When models process a whole video and must account for their own earlier replies, performance falls sharply. One key measure of answer content drops to 61.5% when the system’s conversation history is included. On videos longer than five minutes, models rarely pick the right moment to respond and struggle especially with instructions that change over time.

The examples make the distinction concrete:

While comparison models often only describe what is happening in the video, StreamVibe provides specific next steps. | Image: Google

Source: the-decoder.com

The researchers acknowledge that real situations are likely more complicated than their test conditions. Their next work will examine how user preferences and the surrounding environment affect each other over time.

Google has been building systems that actively analyze video. In September, it introduced agentic video analysis for Gemini, allowing the model to choose which scenes to inspect more closely. More recently, it showed Gemini 4 Argon, which can track long sequences of actions in robot videos and identify errors. StreamVibe adds a different requirement: knowing what is happening at each moment, and whether an intervention would help.

I think the central limitation is less about recognizing an action than maintaining a reliable account of what has happened across time. The benchmark’s own examples point to that gap, while other work cited by the researchers finds models often guess or refuse instead of asking a specific question when visual information is missing. Another video-understanding benchmark found serious weaknesses in spatial and temporal reasoning, and Microsoft researchers showed that conversation itself can impair autonomous agents.

That leaves an unresolved tension: an assistant must remember enough to avoid repeating itself, but also know when its account of the task is incomplete. StreamVibe targets the first problem; the team’s proposed work on preferences and context suggests the second is still open.