SleuthVLM: Do We Really Need Audio-in-Video for Omni Video Understanding?

Abstract

Omni models answer questions about videos by jointly processing sampled frames and audio in one turn, with progress driven largely by scaling model size and omni training data. We ask whether a model needs to hear the audio directly at all, and present SleuthVLM, an agent harness built on a VLM that never hears the audio directly: speech is transcribed into text, and the VLM sleuths the video on its own without any predefined workflow. On four omni benchmarks with videos up to 90 minutes, SleuthVLM achieves the best average accuracy, 75.0%, 2.6 points above Gemini 3.8 Flash, and it also tops the Video-MME-v2 leaderboard with 77.8%; with an open-source 27B VLM, it beats all open-source omni models. We draw three findings: searching outperforms passive observation; agentic ability matters more than direct audio access, as omni training costs part of a VLM’s agentic ability; and small agentic models are highly competitive. So we provide a simple but effective starting point for omni video understanding.

Three findings

Taken together, SleuthVLM provides a simple yet effective starting point for agentic omni video understanding.

Method

SleuthVLM answers audio-visual questions with a vision-language model that never hears the audio directly. The audio is transcribed once per video with word-level timestamps, and the VLM uses a small set of tools to search the video and the transcript. The harness specifies only the accessible resources, a brief operating guide and a budget; the VLM decides where to look, how closely to inspect and when to stop, using the transcript as an index into long videos.

Results

Without hearing the audio directly, SleuthVLM (Opus 5.5) achieves the best average accuracy on four omni benchmarks, ahead of every closed-source omni model, and its lead grows with the length of the video. With an open 27B model, SleuthVLM leads the best open-source omni model by 13.4 points on average and by 18.8 on LVOmniBench.

Video-MME-v2

SleuthVLM (Opus 5.5) also sets a new state of the art on Video-MME-v2: with 77.8% average accuracy and a non-linear score of 63.5, it is the best model on every metric of the leaderboard, 6.8 points above Gemini 3.8 Flash in accuracy. Over the four omni benchmarks and Video-MME-v2 together, it achieves the best average, 75.5%. With Qwen3.8-27B, SleuthVLM reaches 62.0% accuracy and a non-linear score of 44.4, ahead of Gemini-3-Flash, GPT-5 and every open-source model on the leaderboard.

The longer the video, the larger the lead

Case studies

Step-by-step trajectories

BibTeX

@misc{wu2026sleuthvlmreallyneedaudioinvideo,
      title={SleuthVLM: Do We Really Need Audio-in-Video for Omni Video Understanding?},
      author={Peiran Wu and Fangfei Gou and Zhaoxi Mu and Changhao Zhang and Jian Liu and Weiqiang Wang and Yuntao Wang and Yuanchun Shi and Junxiao Shen},
      year={2026},
      eprint={2611.06083},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2611.06083},
}