New Encoder Compresses Text for LLMs Without Fine-Tuning Target Models

Techniques like retrieval-augmented generation and chain-of-thought reasoning have pushed language models to handle longer contexts, but at the cost of higher computational demands. A new method sidesteps the traditional solution of modifying the target model itself.

The Problem with Standard Approaches#

Existing context compression methods typically require fine-tuning or restructuring the language model they're meant to optimize—changes that can hurt the model's general-purpose performance outside that specific use case. Researchers wanted a more portable alternative that wouldn't compromise a model's core capabilities.

How ARC-Encoder Works#

The team developed ARC-Encoder (Adaptable text Representations Compressor), an encoder that transforms input text into condensed continuous representations. These compressed embeddings then replace standard token embeddings within decoder-based large language models. The key insight is that the compression happens outside the target model, avoiding direct modifications to it.

The encoder reduces context to between 4 and 8 times fewer continuous representations than the original token count, significantly cutting inference costs while maintaining performance across multiple evaluation scenarios:

• In-context learning tasks
• Extended context windows
• Both instruction-tuned and base decoder models

Multi-Model Adaptability#

One of ARC-Encoder's strengths is its ability to work with multiple decoders simultaneously. A single pre-trained encoder can generalize across different language models, making it a genuinely portable solution for teams running varied LLM architectures.

Results and Availability#

Testing shows the approach achieves state-of-the-art results on several benchmarks while improving computational efficiency during inference. The researchers have made training code, fine-tuning datasets, and pretrained models publicly available.