Embedding models outweigh chunking strategy in chemistry-focused document retrieval

Researchers have conducted an extensive study examining how different approaches to breaking down and encoding scientific documents affect the performance of retrieval systems used in chemistry question-answering. The work addresses a gap in understanding which technical choices have the most impact when building retrieval-augmented generation (RAG) systems tailored to chemistry literature.

The Challenge of Chemistry Documents#

Scientific documents on chemistry pose specific challenges for information retrieval. They combine specialized terminology with mathematical symbols, quantitative data, and structural formatting that traditional text processing may overlook. This complexity means that generic approaches to organizing and representing documents often fail to capture the nuances needed for accurate evidence retrieval.

Methodology and Scale#

The team developed a custom benchmark called ChemQuests, comprising 952 question-answer pairs drawn from 151 papers published on ChemRxiv across 17 different chemistry subfields. They evaluated 41 embedding models against two existing chemistry retrieval benchmarks (ChemNQRetrieval and ChemHotpotQARetrieval) and then tested the most promising candidates across multiple configurations.

The experimental grid tested:

• Five different chunking strategies
• Seven distinct chunk sizes
• Multiple overlap configurations

Key Findings#

Embedding model choice emerged as the dominant factor in determining retrieval quality. Among the models tested, retrieval-tuned E5, Beijing Academy of Artificial Intelligence General Embedding (BGE), and Nomic embeddings demonstrated the strongest overall performance. These advantages were validated using a geometric-mean metric at rank 10 (Geom@10) against broader retrieval metrics.

For chunking strategy, the results favored a practical middle ground: medium-to-large chunks paired with fixed-token, recursive-token, or hierarchical-section approaches offered the best balance. Additionally, lower overlap between consecutive chunks generally yielded better results across the tested configurations.

Implications for RAG Systems#

The findings suggest that practitioners building chemistry-aware RAG systems should prioritize selecting robust embedding models before fine-tuning document segmentation strategies. While chunking choice does matter, the marginal gains from optimizing segment size and strategy pale compared to the impact of embedding selection, offering a clear roadmap for resource allocation in system development.