Researchers Create Synthetic Datasets to Build AI Systems That Answer Questions in Conversation While Citing Sources

Researchers have developed a method to automatically generate training datasets for conversational question-answering systems that must ground their answers in retrieved documents—a significant challenge given that such labeled datasets are expensive and hard to obtain for specific domains.

The Challenge of Conversational Retrieval#

Building systems that answer questions within a conversation presents unique obstacles. Unlike standalone question answering, these systems must (1) maintain awareness of prior dialog history and (2) provide answers backed by specific documents or passages. Organizations sitting on large repositories of plain-text documents—product manuals, internal wikis, technical guides—lack the annotated training data needed to build such systems in their specific domains.

Synthetic Data Generation Pipeline#

The team's approach capitalizes on abundant organizational documents to automatically create realistic conversational dialogs with the necessary annotations. Their generated datasets include:

• In-dialog question-answer pairs reflecting natural conversational flow
• Decontextualized versions of user questions—standalone, self-contained reformulations that remove pronouns and references to earlier exchanges
• Propositions—key sentences from source documents that system responses should rely on

This mirrors the structure of human-annotated conversational retrieval datasets.

Enabling Existing Retrieval Systems#

A key innovation involves training lightweight question rewriters that strip context from user queries. By converting dialog-dependent questions into self-contained ones, the approach allows organizations to pair off-the-shelf retrievers (which don't understand conversation history) with large language models that generate final responses. The LLM receives both the decontextualized question and the retrieved passages, allowing it to produce grounded answers without requiring retrieval systems to be rebuilt from scratch.

Research Link#

Learn more: arXiv:2507.04884