State Space Models Can Pool Document Representations to Cut Inference Costs

Researchers have discovered that Structured State Space Models (SSMs) can combine representations from multiple documents post hoc without reprocessing the entire input for each new query. This capability, termed document souping, opens a path to more efficient handling of large-scale text reasoning tasks.

How Document Souping Works#

Instead of encoding all documents together in a single pass, the approach encodes each document independently and then pools their hidden states using simple operations like averaging. This modular strategy allows individual document representations to be reused and combined flexibly, creating a unified context state without redundant computation.

Performance Across Key Tasks#

The finetuned Mamba2 models using souped representations achieved competitive or better results than standard monolithic encoding on several benchmarks:

• Multi-hop question answering
• Sparse retrieval tasks
• Long-document reasoning

On the RACE and QuALITY benchmarks for long-document QA, the souping method substantially outperformed traditional concatenation approaches.

Scalability and Efficiency Gains#

The technique scales to hundreds of document segments—the team tested configurations with up to 256 segments—while delivering significant inference cost savings. By avoiding the need to reprocess all input for each query, the method unlocks new possibilities for corpus-scale reasoning applications without proportional increases in computational overhead.