Nvidia combines transcription and speaker ID in Nemotron 3.5

One model, several jobs

Users specify the task with plain-text instructions. The same meeting recording can become a transcript with speaker labels, an English translation or a summary of decisions. Prompts can include specialist terms, such as drug names; the model is supposed to use them only when they were actually spoken.

Nemotron 3.5 Transcribe has 30 billion parameters, with about 4 billion active at each step. Nvidia says it can run on one of its GPUs, from the compact DGX Spark to data-center hardware. On an RTX PRO 6000 Blackwell Server Edition, the company reports, it processes 100 hours of audio in under three minutes at NVFP4 precision.

In Artificial Analysis' comparison, Nemotron 3.5 Transcribe makes barely more errors than Mistral's much larger Voxtral Small while using a fraction of the compute. | Image: Artificial Analysis

Source: the-decoder.com

Nvidia’s reported accuracy results vary by benchmark:

On Hugging Face's Open ASR Leaderboard, Nemotron 3.5 Transcribe edges out open models from Alibaba and IBM. | Image: Hugging Face

Source: the-decoder.com

Speaker identification is where the results look most distinctive. In VoiceArena’s test of 14 systems on conversations with interruptions, Nemotron ranked first, with an 8.32% omission rate against a median of about 31%.

In VoiceArena's comparison, Nemotron 3.5 Transcribe is the best at telling who said what in English conversations where speakers interrupt each other. Lower is better. | Image: VoiceArena

Source: the-decoder.com

On the David AI benchmark for speaker identification across 21 languages, it had the lowest error rate, narrowly beating a separate Nvidia model built for the task.

On David AI's benchmark, Nemotron 3.5 Transcribe is the most reliable at identifying the current speaker across 21 languages. Lower is better. | Image: David AI

Source: the-decoder.com

Data, adaptation and trade-offs

Nvidia is releasing Granary v2 alongside the model: about 844,000 hours of speech across 48 languages, compared with roughly 643,000 hours across 25 languages in the previous version. Each recording has multiple transcript formats: one keeps fillers such as “uh” and spells out numbers, while another removes fillers and uses numerals. Nvidia also publishes the tools used to create the dataset, but says Granary v2 does not fully match the data used to train the model.

The company tested medical-domain adaptation on recordings of Indian doctors and medical students. On the full benchmark, a more specific prompt reduced the error rate from 30.8% to 23.8%; additional training brought it to 11.8%. The training data was predominantly synthetic speech. For the newer speaker model, Nvidia reports a drop in errors from 22.44% to 12.17%, a relative improvement of 45.8%. The adapted model beat an open medical speech recognition model, which scored 13.79%. Nvidia says the process took about 14 hours on one Nvidia GB300 GPU.

The gains come with a trade-off: medical tuning slightly increased errors on ordinary English speech, though it improved recognition of drug brand names. Nvidia recommends setting aside at least 100 real, verified recordings to test custom versions. It also advises spot-checking names, speaker changes and answers to questions about recordings.

I think the more important question is how well the model holds up outside the benchmarks Nvidia highlights. Its capabilities vary by language, its audio question-answering was trained on English, and it handles recorded speech only—not real-time audio. Those limits make the model’s broad task list less decisive than its performance on a user’s own recordings.

A broader audio push

In September, Nvidia introduced Nemotron 3 Diarization, a separate model that can distinguish up to eight speakers, including in real-time conversations. Transcription still required another model, such as Nvidia’s Parakeet. Nemotron 3.5 Transcribe brings speaker identification and transcription together, and adds translation and audio analysis, but gives up real-time processing.

Nvidia has used Granary before, including for its multimodal Nemotron 3 Nano Omni. Mistral’s Voxtral family also combines transcription and audio analysis; its recently announced Voxtral 3 is focused on real-time voice agents.

Nemotron 3.5 Transcribe and Granary v2 are available on Hugging Face, where Nvidia has also posted a demo. The release makes the case for one model covering more of the audio workflow, but the real test is whether that convenience survives the language-specific limits and the loss of live processing.