OmniXRef Is a Unified Framework for Multimodal Video Editing with Multiple References

OmniXRef Unifies Text, Image, and Video References for Multi-Reference Video Editing

Meta description: OmniXRef combines text, image, and video references in one pass, controlling motion, camera, style, visual effects, and content while preserving source videos.

Tags: OmniXRef, video editing, multimodal AI, diffusion transformer, benchmark

Featured image: Abstract multimodal video-editing pipeline with text, image, and video inputs flowing into a single neural network layer, no real people, no company logos, no text.

OmniXRef presents a unified framework for multimodal multi-reference video editing, combining any number of text, image, and video references in a single pass. The value is structural: instead of isolating style, motion control, or camera changes in separate editing stages, the framework treats all references as parts of one editing instruction applied to a source video.

Background: Why Video References Matter

Video editing requests often contain several constraints at once. A user may want a source clip to keep its original appearance while changing the camera path, adding a visual effect, adjusting motion, and applying a new style. Text or images can describe many of these elements, but they are less reliable for temporal detail.

OmniXRef emphasizes video references because intricate visual details and how they change over time, such as camera movement, motion, and visual effects, are hard to specify with text or images alone. This gives the framework a clearer role when the target edit depends on time rather than a single frame.

Development Process: One Interface for Every Reference

The method begins with a unified formulation. Every reference shares one reference-bank interface composed of a task definition, an edit instruction, and an optional image or video. This common interface reduces the mismatch between different input types.

The second step is joint modeling: one interleaved sequence encodes the full editing intent and is processed by a single-stream diffusion transformer (DiT). Instead of running separate conditioners for text, image, and video, the architecture reads all references together and applies them to the source video in one pass.

The training setup is also built from large-scale synthesis. The reported data pipeline yields more than 3 million video-reference editing examples. Those are combined with 7 million text-based editing examples, producing about 10 million examples for unified training. The scale matters because multi-reference editing requires the model to learn how independent attributes interact: a style reference should not necessarily override motion, and a camera reference should not destroy source content.

Latest Progress: OmniXRef-Bench and Reported Results

The project introduces OmniXRef-Bench, a 125-case evaluation suite across motion, camera, style, visual effects, and content. The benchmark measures edit faithfulness, source preservation, and visual quality including temporal consistency. These metrics show what the framework is trying to balance: follow the reference, keep the source intact, and render a coherent moving video.

The reported results say OmniXRef outperforms task-specific baselines and beats Seedance 2.5 and MiniMax-H3 in reference adherence and source preservation. The project page at https://oliver-cong02.github.io/OmniXRef.github.io/ collects the framework, benchmark, and examples.

At this stage, the contribution is a benchmarked unified editing framework rather than a single-task editor. The next signal to watch is how the reference-bank interface behaves when users mix many temporal and visual constraints, and how OmniXRef-Bench results generalize to source videos with more complex motion, effects, and preservation requirements.