Bolting new modalities onto a text embedding model normally costs you the text retrieval you started with, and the usual fix is scale: existing omni-modal embedders "compensate with multi-billion parameters." The deeper problem is that aligning a media encoder to a text space means reconciling two differently-shaped geometries, which is why that literature accumulates calibration machinery — batch whitening, covariance-alignment losses, debiased negative curricula. Omni-Embed-Mini sidesteps all of it by removing the second geometry: text, speech, audio, images, video and visually-rich documents land in one shared cosine space without updating a single text-side parameter.

The trick is that the teacher signal needs no separate embedding model. Each media sample is paired with a dense cascaded caption (generated by Qwen3-Omni-30B-A3B-Instruct, grounded on the source dataset's own ground-truth caption), and the training target is simply the frozen backbone's own embedding of that caption. Teacher and student share the same weights, so they "inhabit byte-identical geometry" and the space being aligned to is already calibrated and never moves. What trains is lightweight projectors plus phased LoRA on the modality encoders: projectors only for the first 20% of steps, then encoder LoRA (r=16, α=32) switches on, so the adapters never adapt to noisy upstream projections. The objective is a Matryoshka SigLIP contrastive loss with an online hybrid hard-negative miner whose negatives sharpen as the encoder improves. The 0.9B build freezes Qwen3-Embedding-0.6B and attaches a ViT extracted from Qwen3.5-0.8B plus parallel Whisper-small and Dasheng-base audio encoders at 128 projected tokens each — 68.32M of 935.35M parameters trained (7.3%); the 2.3B build swaps in a frozen Qwen3-VL-Embedding-2B and trains 141.68M (5.8%). Omni-Sets supplies 590,858 rows, 227,704 effective after duration filtering (~60K speech, ~50K audio, ~80K image, ~36K video, remainder visual documents).

The frozen anchor delivers what it promises: the 0.9B scores 49.57 nDCG@10 on MTEB-v2 BEIR-8 with its text path untrained, second among open omni embedders only to omni-embed-nemotron-3B (50.48) at five times the size, while being 2.7–9.5× smaller than every open omni embedder compared. The 2.3B variant reaches an overall-modality average of 51.39 against 49.51 for the closed gemini-embedding-2, leading on image (64.80), video (55.18) and visual documents (58.10) while trailing it on text, speech and audio. The paper is explicit about where this stops: e5-omni-7B still leads the overall average at 52.93, and at 0.9B image (26.29) and video (18.48) are weak because the extracted ViT came from a 0.8B backbone and saw only ~86K image and ~36K video rows, with speech and audio trailing the larger omni baselines — "the expected cost of adding these modalities through small projectors rather than by retraining the backbone."

Two ablations carry the argument. Dense cascaded captions versus the source datasets' original captions, on the 0.9B mining-ablation checkpoint, is worth +14.06 nDCG@10 on video, +8.60 image, +2.70 audio, +2.68 speech (+7.01 overall) — so the caption density, not the architecture, is doing the work. And unfreezing backbone LoRA at 0.9B does improve media by +1.10 to +5.66 while costing 3.13 points of text, which is precisely the regression the frozen-anchor design exists to prevent. Combined text-plus-media hard-negative mining is best overall (+2.6 mean over media), though visual documents are near-insensitive to the mining strategy (all checkpoints within 0.6 nDCG@10). Stated limitations: the self-distillation pipeline inherits whatever its teacher captioner hallucinates or omits, with no controlled caption-perturbation study; the audio pathway uses fixed temporal interleaving of the two encoders, "suboptimal for inputs dominated by only one audio type"; and video evaluation is capped by per-frame mean-pooling. Published in Findings of EMNLP 2026. Training ran on 8 GPUs under DDP in BF16; the 2.3B checkpoint on HuggingFace is gated.

Model Details

Architecture DENSE

Benchmark Scores

Benchmark Score Mode
MTEB-v2 BEIR-8 (nDCG@10, 0.9B) 49.57 —
Overall-modality average (2.3B) 51.39 —
MMEB-V2 Video (2.3B) 55.18 —
MMEB-V2 Image (2.3B) 64.80 —
ViDoRe-V3 (2.3B) 58.10 —

Variants

Name Parameters Notes
Omni-Embed-Mini-0.9B — frozen Qwen3-Embedding-0.6B backbone; 68.32M of 935.35M params trained
Omni-Embed-Mini-2.3B — frozen Qwen3-VL-Embedding-2B backbone; 141.68M of 2.44B params trained

Paper

Venue Findings of EMNLP 2026
Authors: Mohammed Irfan Kurpath · Jaseel Muhammad Kaithakkodan · Sahal Shaji Mullappilly · Ivan Laptev · Hisham Cholakkal
multimodalembeddingsretrievalefficiencydistillationopen-weightresearch

Related