AI System Generates Research Ideas from 58M Papers, but Expert Interest Remains Limited

Researchers have built a system called SciMuse that taps into an enormous corpus of scientific literature to propose novel research directions tailored to individual investigators. Despite the scale of the effort—drawing on data from 58 million papers and deploying a large language model to synthesize ideas—the results reveal a sobering gap between what AI can generate and what domain experts actually find compelling.

How SciMuse Works#

SciMuse operates by leveraging a structured knowledge graph built from decades of published research alongside modern language models. The system generates personalized research ideas by considering an investigator's prior work and finding promising intersections with other fields and concepts. The approach is meant to help researchers navigate an overwhelming volume of scientific output to surface genuinely novel directions.

Expert Evaluation Results#

To assess whether the tool produces genuinely interesting ideas, the team conducted an ambitious evaluation involving more than 100 research group leaders across disciplines ranging from the natural sciences to the humanities. These experts rated over 4,400 customized ideas on a five-point scale of interest. The findings were modest: the average rating landed at 2.40, and the most frequent score was 1—indicating limited enthusiasm. Only about one in four ideas (24.9%) scored 4 or 5, suggesting true acceptance remains rare.

Knowledge Graph Strategy Underperforms#

Counterintuitively, the knowledge graph approach did not enhance perceived quality. When researchers supplied concept pairs selected using graph-based methods, expert ratings did not improve compared to a simpler baseline that only examined paper titles combined with GPT. More striking still, concept pairs predicted to be high-impact based on citation patterns actually trended lower in perceived interest than randomly selected pairs—a statistically meaningful finding that challenges intuitive assumptions about what makes research appealing.

Predicting and Steering Interest#

Despite the modest ratings overall, the researchers identified a path forward. Graph-based features proved useful for controlling and predicting which ideas experts would find more appealing. Both supervised neural networks trained on graph features and zero-shot LLM-based ranking approaches could forecast expert interest with some accuracy. This suggests the underlying data carries signal—even if capturing it remains difficult.