ECCV 2026

Video-Oasis: Rethinking Evaluation of Video Understanding

Geuntaek Lim^1,★, Sungjune Park¹, Jaeyun Lee¹, Inwoong Lee², Taeoh Kim², Dongyoon Wee²
Minho Shim^2,†, Yukyung Choi^1,†

¹Sejong University · ²NAVER Cloud

^★ Work done during an internship at NAVER Cloud ^† Co-corresponding authors

The audit, in one bar 24,416 QA samples · 14 benchmarks

Shortcut-solvable ≈ 55%

Video-native 45%

Answerable without genuine video understanding Requires visual evidence + temporal context → Video-Oasis (11,033 QA)

14video benchmarks audited under shared criteria

≈ 55%of samples solvable through non-video shortcuts

46.7%best model (Gemini-2.5 Pro) on the distilled challenges · chance 25.63%

TL;DR

Video-Oasis audits whether today's video benchmarks truly measure video understanding. Using shared visual and temporal criteria across 14 benchmarks, we find that about 55% of QA samples are solvable through non-video shortcuts — and we distill what remains into five video-native challenges where most current models perform near random chance.

Motivation

What makes a benchmark truly measure video understanding?

Criterion 01 · Visual grounding Answers should depend on grounded visual evidence from the raw video.

Criterion 02 · Temporal reasoning Answers should depend on temporal context, not isolated frames alone.

Video benchmarks are rapidly expanding, but high scores do not always imply video-native understanding. Many video-QA samples can be answered through linguistic priors, transcript reliance, single-frame perception, or static context. Below, we expose these shortcut pathways through concrete cases — click any card to inspect the full question.

Shortcut pathways in video understanding

Arithmetic ReasoningCase 01

Emily bought stock at $50/share in 2005 and passed away in 2020 at $150/share. Her heir sold it in 2023 at $180/share. Under the “step-up in basis” rule, how much capital gains tax is due?

Why shortcut? Text-only math — the video is never needed.

Video-MMEOpen case →

Single FrameCase 02

What color is the stationary cube when the video begins?

Why shortcut? One frame reveals the answer; motion is irrelevant.

MVBench▶ clip

World KnowledgeCase 03

According to the video, how many grandsons does the founder of Gucci have?

Why shortcut? External knowledge about Gucci suffices.

Video-MMEOpen case →

Static ContextCase 04

What appears in the current scene?

Why shortcut? Any representative frame shows the scene content.

RTV-Bench▶ clip

Linguistic ReasoningCase 05

How does the speaker think about David Suchet’s version of the movie?

Why shortcut? The transcript alone answers it; pixels are optional.

Video-MMEOpen case →

Implicit BiasCase 06

What causes the man in the red hat to shrink?

Why shortcut? Super Mario priors give the answer away.

ImplicitQA▶ clip

Method

What is Video-Oasis?

Video-Oasis rethinks the current benchmark landscape by using shared criteria to audit whether proliferating video benchmarks truly require visual evidence, temporal context, and reliable annotations for genuine video understanding.

01 Visual

Determine whether the task truly requires grounded visual perception rather than language, audio, or text-based cues.

02 Temporal

Determine whether the answer depends on temporal order and context rather than isolated or unordered frames.

03 Ambiguity

Ensure that questions have reliable annotations, unique answers, and evidence anchored to the relevant video context.

Overview of the Video-Oasis diagnostic suite.

Finding

Hidden shortcuts behind high scores

We estimate how often benchmark samples remain solvable without visual evidence or temporal context, and compare these shortcut ratios with reported accuracy. High scores often reflect shortcut-solvable examples rather than video-native understanding.

EgoSchema ImplicitQA VSI-Bench TVBench VCR-Bench RTV-Bench Video-Holmes MINERVA MMR-V Video-MME MVBench LVBench LongVideoBench MLVU

Benchmarks analyzed

Coverage

Spatial Temporal Reasoning General

Findings

High scores go hand in hand with high shortcut rates.
≈ 55% of benchmark samples are shortcut-solvable.

Benchmark accuracy and shortcut ratio across video understanding benchmarks. — Reported accuracy vs. shortcut-solvable ratio across the 14 audited benchmarks.

What are video-native challenges?

Video-native challenges are the non-trivial cases that remain after shortcut pathways are removed, requiring models to rely on visual evidence, temporal context, and grounded spatio-temporal reasoning.

Beyond shortcuts They cannot be explained by linguistic priors, transcripts, static context, or single-frame evidence alone.

Data-driven distillation Five recurring challenge categories emerge by distilling existing benchmarks after shortcut filtering.

Diagnostic testbed They offer a controlled setting for analyzing model improvements under strict visual and temporal dependencies.

Overview of the five video-native challenges distilled by Video-Oasis. — The five video-native challenges distilled from 14 benchmarks after shortcut filtering.

Fine-Grained Perception

Ground fine-grained visual recognition in how details evolve across space and time.

View example →

Spatial World Understanding

Synthesize fragmented, multi-view evidence across frames to infer 3D positions, geometry, and trajectories.

View example →

Temporal Dynamics & Tracking

Track objects, action sequences, and state transitions in their correct temporal order.

View example →

Causality & Logical Reasoning

Infer cause-and-effect relationships, physical rules, and unobserved intentions beyond visible events.

View example →

Global Narrative

Aggregate events across the full timeline, filtering irrelevant context to understand overarching plots and long-term semantics.

View example →

Evaluation

How do state-of-the-art models perform?

We evaluate a broad range of models on the Video-Oasis-distilled challenges and summarize the main trends below.

Near-chance performance. Many current Video-LLMs remain close to the random-chance level.
Current frontier. Gemini-2.5 Pro performs best, but the tasks remain non-trivial even for advanced proprietary models.
Holistic bottleneck. Global Narrative remains a persistent bottleneck for long-term multi-scene understanding.
Agentic design matters. The gap between VideoTree and STAR highlights the impact of reasoning-step orchestration.

01 · Video-Native Challenges Random chance: 25.63%

Model	Fine-Grained Perception	Spatial World	Temporal Dynamics	Causal Reasoning	Global Narrative	Overall
Proprietary LLMs
GPT-4o	25.6	33.2	26.3	27.3	26.5	27.5
Gemini-2.5 Pro	40.2	49.8	50.9	45.4	43.0	46.7
Open-Source Video-LLMs
Qwen2.5-VL (7B)	23.3	28.7	32.3	28.6	21.2	29.2
Qwen3-VL Inst. (8B)	27.0	42.4	36.5	28.0	21.5	33.8
Qwen3-VL Think. (8B)	29.0	41.6	37.7	27.7	23.2	34.6
Eagle2.5 (8B)	26.9	31.0	39.7	33.2	22.7	34.5
InternVL-3 (8B)	27.0	31.3	34.1	30.6	24.5	31.6
InternVL-3.5 (8B)	29.5	41.9	35.1	29.8	23.3	33.6
Video-R1 (7B)	24.0	24.0	29.1	27.3	18.4	26.3
LongViLA-R1 (7B)	28.4	25.4	31.5	27.9	20.6	28.6
VideoAuto-R1 (8B)	27.5	44.3	39.5	31.1	28.9	36.8
Agentic Methods
VideoTree (GPT-5 mini)	28.6	34.0	32.3	24.6	20.7	30.1
STAR (GPT-5 mini)	31.6	44.4	42.2	34.0	32.9	39.5

02 · Evaluation Gap

Video-native challenges require strict visual evidence and temporal context, where many models perform close to random chance. These results indicate that current benchmarks can overestimate models’ video understanding capabilities.

Performance collapse after filtering shortcut-solvable benchmark samples. — Model accuracy recomputed after removing samples solvable through non-video shortcuts.

Analysis

Algorithmic designs from video-native challenges

Because video-native challenges emphasize visual and temporal dependencies, they expose where current design choices help, where they remain limited, and which directions deserve closer investigation.

How much does temporal grounding matter?

QuestionDo video-native challenges truly require strict spatio-temporal dependency?

Practical temporal grounding (with AKS)

Method	Grounding	Overall
Eagle2.5	–	31.5
Eagle2.5	✓	32.9 (+1.4)
Qwen3-VL Inst.	–	27.8
Qwen3-VL Inst.	✓	30.1 (+2.3)

Oracle temporal grounding (ground-truth region)

Method	Video-Oasis	Shortcut
Eagle2.5	35.0	78.0
Eagle2.5 + Oracle	50.8 (+15.8)	80.8 (+2.8)

FindingsPrecise grounding becomes increasingly important when strong spatio-temporal dependencies are required — the oracle gain appears almost exclusively on video-native samples.

Should models always think?

QuestionDoes choosing the right reasoning mode improve video-native understanding?

Reasoning depth modulation

Method	Overall
Qwen3-VL Inst.	33.8
Qwen3-VL Think.	34.6
VideoAuto-R1	36.8
Qwen3-VL Inst. or Think. (oracle)	46.2 (+9.4)
Gemini-2.5 Pro	46.7

FindingsStrategic optimization of when to think can be as impactful as raw model scale.

What works — and what remains — for training?

QuestionDo SFT and RLVR provide a single best route, or complementary strengths?

Training paradigms (base model: Qwen2.5-VL)

Paradigm	Model	Reward	Overall
SFT	Qwen2.5-VL	–	29.2
SFT	Eagle2.5	–	34.5 (+5.3)
RLVR	Video-R1	QA	26.3 (−2.9)
RLVR	VideoAuto-R1	QA + Grounding	32.7 (+3.5)

Findings

Long-context SFT helps: Eagle2.5 improves over Qwen2.5-VL without RLVR.
Reward design matters: RLVR outcomes vary substantially (Video-R1 < Qwen2.5-VL < VideoAuto-R1).
Strengths are complementary: SFT lifts overall accuracy, while RLVR can help hard tasks such as Global Narrative (21.2 → 28.6).

Citation

BibTeX

@inproceedings{lim2026videooasis,
  title     = {Video-Oasis: Rethinking Evaluation of Video Understanding},
  author    = {Lim, Geuntaek and Park, Sungjune and Lee, Jaeyun and Lee, Inwoong
               and Kim, Taeoh and Wee, Dongyoon and Shim, Minho and Choi, Yukyung},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026}
}