AI
New Framework Measures LLM Creativity Across Multiple Domains and Dimensions
Researchers present CreativityPrism, a scalable evaluation system that tests language models on divergent thinking, creative writing, and logical reasoning to reveal gaps in how AI systems approach different types of creative tasks.
Assessing whether large language models can truly be creative has been challenging due to fragmented evaluation methods and the reliance on manual human assessment. Researchers have now introduced CreativityPrism, a unified framework designed to measure creativity in LLMs across diverse contexts using automated, scalable evaluation metrics.
What CreativityPrism Covers#
The framework consolidates eight tasks spanning three domains: divergent thinking (generating multiple solutions), creative writing, and logical reasoning. Rather than viewing creativity as a single quality, the system evaluates it through three distinct dimensions: quality (how good the output is), novelty (how original or unexpected), and diversity (how varied the responses are). This multi-dimensional approach allows for a more nuanced picture of where different models excel.
Key Findings from the Analysis#
Testing 17 state-of-the-art language models revealed significant disparities in performance. Frontier-scale models showed a substantial edge—roughly 15%—over locally-deployable open-source alternatives in creative writing and logical reasoning. However, the advantage disappeared in divergent thinking tasks, a domain that has received far less attention in model training pipelines.
Perhaps more striking was the lack of transfer: strong performance in one creative dimension rarely predicted success in another. Novelty metrics in particular showed weak or even negative correlations with other performance measures, suggesting that simply making text more original can sometimes come at the cost of other creative qualities.
Why This Matters#
The framework's automated judges have been validated against human assessments, making it both faster and more scalable than purely human-led evaluation. This addresses a longstanding bottleneck in understanding LLM capabilities. The findings confirm that creativity itself is multi-faceted—success on one task or dimension does not guarantee generalization across the creative landscape, indicating that evaluating and improving LLM creativity requires targeted, domain-specific approaches rather than blanket improvements.
