AI
New Method Turns LLMs Into Interpretable Text Embeddings While Preserving Safety Features
Researchers present LLM2Vec-Gen, a self-supervised approach that generates embeddings directly from language models while keeping their safety and reasoning capabilities intact.
Researchers have proposed a novel approach to extract embeddings from large language models that retains key properties of the underlying models while achieving state-of-the-art performance on standard benchmarks.
The Challenge with Traditional Embedding Methods#
Most current techniques for fine-tuning LLM-based text embedders rely on contrastive learning, which creates a new representational space and discards the original model's semantic information in the process. This transformation can compromise beneficial qualities that the LLM naturally possesses, such as safety safeguards and reasoning abilities.
How LLM2Vec-Gen Works#
The new method, called LLM2Vec-Gen, takes a fundamentally different approach by operating directly within the LLM's native output space. Instead of projecting into an entirely new representation, the technique appends trainable special tokens to input text and optimizes them to compress the model's own response into a fixed-length embedding.
Key features of the approach include:
- Frozen backbone: The underlying LLM remains unchanged during training
- Unsupervised learning: Training requires only unlabeled queries, with guidance from an embedding teacher and a reconstruction objective
- Interpretability: The resulting embeddings can be decoded back into text, making their semantic content transparent
Performance and Benefits#
LLM2Vec-Gen achieved top self-supervised results on the Massive Text Embedding Benchmark (MTEB), improving by 8.8% compared to its unsupervised embedding teacher.
Beyond raw performance, the method preserves critical LLM properties:
- Safety alignment: Achieves up to 22.6% reduction in harmful content retrieval
- Reasoning capability: Shows up to 35.6% improvement on reasoning-intensive retrieval tasks
These gains emerge naturally because the embeddings maintain the semantic space of the original model's responses, rather than transforming into an entirely different representation.
