Omni-InteractScope

Benchmarking Proactive and Agentic Interaction
in Continuous Omnimodal Streams

Yinsong Yan1,2, Ling Wang1,2, Yuxuan Wang2, Yifan Yang2,3, Ruiyang Xu2,3,
Junming Lin2,4, Qi Chen2,3, Meng Gao2,4, Minghao Han2,5, Yunfei Chu2,
Qize Yang2, Xize Cheng2, Jin Xu2, Jibin Wu1

1The Hong Kong Polytechnic University   2Alibaba Token Hub, Alibaba Group
3Shanghai Jiao Tong University   4Tsinghua University   5Fudan University

Overview

Omni-InteractScope evaluates assistants that follow continuous video and audio streams and respond at useful moments. Each interaction combines a stream with user context in text, speech, or images. Responses may be answers, alerts, warnings, tool calls, or successive guidance; the benchmark evaluates their content, timing, and coverage.

The benchmark contains 1,479 interactions and 1,790 response targets across eight task groups. T1–T7 use online videos spanning indoor, outdoor, and digital-screen scenes to test explicit requests. T8 uses self-recorded first-person videos for visual and hearing assistance, including reminders without an event-specific question, audiovisual questions, and multi-step guidance.

8 task groups1,479 interactions1,790 response targets12 model configurations

Tasks

Tasks T1–T7 begin with an explicit request. T8 covers assistance through reminders, audiovisual questions, and step-by-step guidance.

T1Tool invocationInvoke a tool when the requested event occurs.
T2Event notificationNotify the user when a specified event appears.
T3Risk and error interjectionIntervene when a hazard or mistake becomes apparent.
T4Cumulative-state trackingFollow a count or state as it changes over time.
T5Temporal conditionsRespond when a condition about time is met.
T6Nonverbal and workflow cuesUse gestures and context to infer when help is needed.
T7Continuous responsesGive multiple timely responses as events unfold.
T8AssistanceProvide reminders (A), answer audiovisual questions (B), or guide a multi-step task (C).

Results

Results for all 12 model configurations across the eight task groups.

Evaluation on explicit-request tasks (T1–T7)

Joint content–timing accuracy (%), higher is better. Avg. equally weights T1–T4, T6, and T7.

Model Size Input T1 T2 T3 T4 T5† T6 T7 Avg.
Proprietary models
Gemini-3.1-Pro N/R V, A 22.5 28.2 11.6 11.2 N/A 27.0 11.2 18.6
Gemini-3.5-Flash N/R V, A 29.5 31.6 14.0 10.2 N/A 24.3 11.4 20.2
Qwen3.5-Omni-Plus N/R V, A 32.0 24.3 4.8 3.9 N/A 10.8 7.9 13.9
Open-source models
InternVL3.5 8B V+ASR 29.0 1.0 0.0 0.0 N/A 5.4 0.4 6.0
Qwen3-VL 8B V+ASR 19.5 4.9 0.5 0.0 N/A 2.7 3.1 5.1
Qwen2.5-Omni 7B V, A 11.0 0.5 0.5 0.5 N/A 0.0 1.5 2.3
Qwen3-Omni 30B-A3B V, A 26.5 2.9 1.4 1.0 N/A 2.7 5.7 6.7
Streaming Models
LiveCC 7B V 0.0 1.5 0.5 0.0 0.0 0.0 0.7 0.4
AURA 8B V 0.0 4.9 0.5 1.5 0.0 2.7 0.7 1.7
MMDuet2 3B V 0.0 3.4 1.0 0.5 0.0 2.7 1.8 1.6
MOSS-VL 11.3B V 1.5 9.2 1.9 5.8 0.9 8.1 5.9 5.4
MiniCPM-o 4.5 9B V, A 1.5 8.3 1.4 1.0 1.7 2.7 1.1 2.7

† T5 uses a separate query-relative timing protocol and is excluded from Avg. N/A means no explicit elapsed-time clock.

Evaluation on assistance tasks (T8)

Timely reference-target recall (%), higher is better. Overall pools 360 targets: 122 for T8-A, 151 for T8-B, and 87 for T8-C.

Model Size Input Overall T8-A T8-B T8-C Complete T8-C
Proprietary models
Gemini-3.1-Pro N/R V, A 46.39 22.95 62.25 51.72 3/26
Gemini-3.5-Flash N/R V, A 41.11 22.13 57.62 39.08 2/26
Qwen3.5-Omni-Plus N/R V, A 31.11 9.02 49.67 29.89 1/26
Open-source models
InternVL3.5 8B V+ASR 14.17 0.00 26.49 12.64 0/26
Qwen3-VL 8B V+ASR 15.00 2.46 24.50 16.09 0/26
Qwen2.5-Omni 7B V, A 10.00 2.46 19.87 3.45 0/26
Qwen3-Omni 30B-A3B V, A 13.06 2.46 25.83 5.75 0/26
Streaming Models
LiveCC 7B V 3.61 0.00 8.61 0.00 0/26
AURA 8B V 6.94 0.00 15.89 1.15 0/26
MMDuet2 3B V 1.94 0.00 4.64 0.00 0/26
MOSS-VL 11.3B V 3.33 0.00 7.28 1.15 0/26
MiniCPM-o 4.5 9B V, A 3.06 0.00 7.28 0.00 0/26

Complete T8-C counts guidance interactions with every step covered, out of 26.

Input: V = video, A = native audio, ASR = scene transcripts. N/R = undisclosed model size; A3B = active parameters. The first seven models use independent prompted calls; streaming models use rolling evaluation. Scoring protocol.

Examples

T8-A · Query-free reminderPark walkway

A group of chickens moves onto the park walkway. Without waiting for an event-specific question, the assistant should warn the user before they reach the birds.

26.8 s · “There are chickens on the path ahead. Slow down and walk around them.”

T8-B · Audiovisual questionPet shop

User: “Did this black-and-white bird chirp?” Bird calls are audible while several birds are visible, but the sound cannot be attributed to the bird in question.

5.7 s · “I hear birds chirping, but the white bird with black spots is feeding, so I cannot tell whether it made the sound.”

T8-C · Multi-step guidanceMetro gate

User: “I need to use a single-journey ticket to enter the metro station through the entry gate.” The assistant guides the user through the gate in three steps.

  1. 3.3 s · Walk toward the entry gates ahead.
  2. 8.9 s · Scan the ticket on the reader to the right.
  3. 11.9 s · After the successful scan, pass through the gate.