Unraveling Textual Chaos: Exposing Modality Bias in Mixed-Modality Retrievers

Trident Builds Three Document Views to Fix Mixed-Modality Retrieval

Meta description: Trident tackles mixed-modality retrieval bias as dense retrievers struggle when text, image, and fused documents coexist, exposing Chaos in the Text.

Tags: dense retrievers, mixed-modality retrieval, Trident, multimodal search, document AI

Featured image: An abstract technology scene with overlapping text panels, image tiles, and fused document nodes; no real people, no company logos, no text.

A paper on mixed-modality retrieval reports that dense retrievers, the vector-based engines used to find relevant content, can lose accuracy when text, image, and fused text-image documents are mixed. The proposed method, Trident, builds three views for each document and uses them to reduce a failure the study calls Chaos in the Text. The result matters for any system that must rank both written content and visual content from the same corpus.

Background: Dense Retrievers Meet Mixed Corpora

Dense retrievers encode content into fixed-size representations and rank documents by comparing query and document vectors. That design has worked well on text corpora and on image corpora, where the training signal is usually concentrated in one modality. The open question is not whether these retrievers can handle a pure text collection or a pure image collection. It is whether they remain reliable when a corpus contains text, image, and fused text-image documents at the same time. Mixed corpora make that question harder because the model can no longer treat one signal as the default. A screenshot, a paragraph, and a fused text-image page may all be relevant to the same query, but they are represented in different ways.

Development: A V-Shaped Curve Reveals Chaos in the Text

The study's central experiment changes the corpus gradually. Image documents are progressively replaced with semantically corresponding text representations. The resulting retrieval behavior does not follow a smooth decline. It forms a pronounced V-shaped curve. At the single-modality ends, performance remains strong. In the mixed middle, it degrades, showing that the retriever is sensitive to modality composition rather than only to semantic relevance.

The most pointed finding is about distractors. When irrelevant text is added, retrieval quality falls more severely than when an equal number of irrelevant images is added. The study names this failure Chaos in the Text. The analysis explains why: the retriever shows a modality preference, assigning systematically higher similarity scores to text representations. That preference lets irrelevant text outrank relevant images, turning the mixed corpus into a trap for otherwise capable models.

Latest Progress: Trident Turns Each Document into Three Co-Equal Views

Trident addresses the bias by changing how each document is represented during training. The method constructs three views of each document: a text view, an image view, and a fused text-image view. Those views are treated as co-equal positives, not as one primary representation with auxiliary copies. The training objective, Multi-Positive View InfoNCE, optimizes two goals together: relevance discrimination, which helps the model separate relevant from irrelevant content, and positive-view balance, which keeps the different views from pulling the retriever toward one modality.

Across visual document and natural image benchmarks, the paper reports that this setup improves mixed-modality retrieval on both CLIP-based and VLM-based architectures. It also reduces sensitivity to modality composition and text distractors. Importantly, the gain is not framed as a trade-off that weakens unimodal use: the study says Trident increases average single-modality retrieval performance.

The current stage is a diagnostic and mitigation phase: mixed-modality retrieval has been separated into a named failure mode, and Trident tests a three-view correction on two architecture families and two benchmark categories. The next question is whether the same three-view setup holds beyond visual document and natural image benchmarks, and across more retriever families.