Google AI Introduces Gemini Embedding 2: A Multimodal Embedding Model that Lets Your Bring Text, Images, Video, Audio, and Docs into the Embedding Space

Google expanded its Gemini model family with the release of

Gemini Embedding 2

. This second-generation model succeeds the text-only

gemini-embedding-001

and is designed specifically to address the high-dimensional storage and cross-modal retrieval challenges faced by AI developers building production-grade

Retrieval-Augmented Generation (RAG)

systems. The

Gemini Embedding 2

release marks a significant technical shift in how embedding models are architected, moving away from modality-specific pipelines toward a unified, natively multimodal latent space.

Native Multimodality and Interleaved Inputs

The primary architectural advancement in Gemini Embedding 2 is its ability to map

five

distinct media types—

Text, Image, Video, Audio, and PDF

—into a single, high-dimensional vector space. This eliminates the need for complex pipelines that previously required separate models for different data types, such as CLIP for images and BERT-based models for text.

The model supports

interleaved inputs

, allowing developers to combine different modalities in a single embedding request. This is particularly relevant for use cases where text alone does not provide sufficient context.

The technical limits for these inputs are defined as:

• 
  
  Text:
  
  Up to 8,192 tokens per request.
  


• 
  
  Images:
  
  Up to 6 images (PNG, JPEG, WebP, HEIC/HEIF).
  


• 
  
  Video:
  
  Up to 120 seconds of video (MP4, MOV, etc.).
  


• 
  
  Audio:
  
  Up to 80 seconds of native audio (MP3, WAV, etc.) without requiring a separate transcription step.
  


• 
  
  Documents:
  
  Up to 6 pages of PDF files.
  

By processing these inputs natively, Gemini Embedding 2 captures the semantic relationships between a visual frame in a video and the spoken dialogue in an audio track, projecting them as a single vector that can be compared against text queries using standard distance metrics like

Cosine Similarity

.

Efficiency via Matryoshka Representation Learning (MRL)

Storage and compute costs are often the primary bottlenecks in large-scale vector search. To mitigate this, Gemini Embedding 2 implements

Matryoshka Representation Learning (MRL)

.

Standard embedding models distribute semantic information evenly across all dimensions. If a developer truncates a 3,072-dimension vector to 768 dimensions, the accuracy typically collapses because the information is lost. In contrast, Gemini Embedding 2 is trained to pack the most critical semantic information into the earliest dimensions of the vector.

The model defaults to

3,072 dimensions

, but

Google team has optimized three specific tiers for production use:

1. 
   
   3,072:
   
   Maximum precision for complex legal, medical, or technical datasets.
   


2. 
   
   1,536:
   
   A balance of performance and storage efficiency.
   


3. 
   
   768:
   
   Optimized for low-latency retrieval and reduced memory footprint.
   

Matryoshka Representation Learning

(MRL) enables a ‘short-listing’ architecture. A system can perform a coarse, high-speed search across millions of items using the 768-dimension sub-vectors, then perform a precise re-ranking of the top results using the full 3,072-dimension embeddings. This reduces the computational overhead of the initial retrieval stage without sacrificing the final accuracy of the RAG pipeline.

Benchmarking: MTEB and Long-Context Retrieval

Google AI’s internal evaluation and performance on the

Massive Text Embedding Benchmark (MTEB)

indicate that Gemini Embedding 2 outperforms its predecessor in

two specific areas

:

Retrieval Accuracy

and

Robustness to Domain Shift

.

Many embedding models suffer from ‘domain drift,’ where accuracy drops when moving from generic training data (like Wikipedia) to specialized domains (like proprietary codebases). Gemini Embedding 2 utilized a multi-stage training process involving diverse datasets to ensure higher zero-shot performance across specialized tasks.

The model’s

8,192-token window

is a critical specification for RAG. It allows for the embedding of larger ‘chunks’ of text, which preserves the context necessary for resolving coreferences and long-range dependencies within a document. This reduces the likelihood of ‘context fragmentation,’ a common issue where a retrieved chunk lacks the information needed for the LLM to generate a coherent answer.

Key Takeaways

1. 
   
   Native Multimodality
   
   : Gemini Embedding 2 supports five distinct media types—
   
   Text, Image, Video, Audio, and PDF
   
   —within a unified vector space. This allows for
   
   interleaved inputs
   
   (e.g., an image combined with a text caption) to be processed as a single embedding without separate model pipelines.
   


2. 
   
   Matryoshka Representation Learning (MRL)
   
   : The model is architected to store the most critical semantic information in the early dimensions of a vector. While it defaults to
   
   3,072 dimensions
   
   , it supports efficient truncation to
   
   1,536
   
   or
   
   768
   
   dimensions with minimal loss in accuracy, reducing storage costs and increasing retrieval speed.
   


3. 
   
   Expanded Context and Performance
   
   : The model features an
   
   8,192-token input window
   
   , allowing for larger text ‘chunks’ in RAG pipelines. It shows significant performance improvements on the
   
   Massive Text Embedding Benchmark (MTEB)
   
   , specifically in retrieval accuracy and handling specialized domains like code or technical documentation.
   


4. 
   
   Task-Specific Optimization
   
   : Developers can use
   
task_type

   parameters (such as
   
RETRIEVAL_QUERY

   ,
   
RETRIEVAL_DOCUMENT

   , or
   
CLASSIFICATION

   ) to provide hints to the model. This optimizes the vector’s mathematical properties for the specific operation, improving the “hit rate” in semantic search.
   

Check out


Technical details


, in Public Preview via the


Gemini API


and


Vertex AI



.

Also, feel free to follow us on



Twitter



and don’t forget to join our


120k+ ML SubReddit


and Subscribe to


our Newsletter


. Wait! are you on telegram?


now you can join us on telegram as well.