Embedding-Based RAG Beats GraphRAG for Math Textbook Question Answering

Researchers testing how large language models can serve as personalized tutors have found that straightforward embedding-based retrieval works better than more sophisticated graph-based approaches for answering questions tied to specific textbook sections.

The team evaluated multiple strategies for grounding AI responses in course materials using a curated dataset of 477 question-answer pairs from an undergraduate mathematics textbook, where each question mapped to a single source page. They measured both whether the system retrieved the correct page and the quality of the generated answers.

Key findings#

The results revealed a clear winner among the retrieval methods tested:

• Voyage-3-large, an embedding-based model, achieved 99.4% accuracy at top-10 retrieval and 64–73% accuracy for the most challenging top-1 metric (95% confidence interval)
• BM25, a keyword-search baseline from decades past, performed better than several modern embedding models—an unexpected strength that suggests traditional approaches remain competitive
• GraphRAG, which constructs knowledge graphs to capture document relationships, underperformed at the retrieval stage

A critical weakness of GraphRAG became apparent during error analysis: it returned far more context than necessary. While embedding-based RAG provided roughly 3,700 tokens per query, GraphRAG delivered around 47,000—excess information that degraded the quality of final answers from the language model.

Lessons for weaker models#

The researchers replicated their key experiments using an open-source local language model (Qwen 3.5-35B) that was less capable than larger commercial systems. This revealed an important pattern: as the underlying LLM gets weaker, the benefit of good retrieval grows disproportionately. RAG improved answer quality by 39% for the local model versus only 16% for stronger systems—suggesting that retrieval-augmented approaches are especially valuable in cost-conscious educational deployments where powerful models may not be feasible.

Why it matters for education#

The findings have direct implications for building AI tutoring systems that need to cite specific course materials. Even when retrieval failed, the system typically returned pages from the same chapter as the correct answer, preserving some pedagogical relevance. This suggests room to improve performance through smarter ranking or filtering strategies tailored to educational contexts.