New benchmark exposes major gaps in AI's ability to build 3D scenes from text descriptions

Generating 3D scenes from written descriptions is increasingly important for embodied AI systems and virtual environments, yet assessing how well these systems actually match user intent remains a stubborn challenge. Researchers have now unveiled a new evaluation framework that goes far beyond existing methods by checking whether synthesized scenes truly respect the specific constraints hidden within text descriptions.

The problem with current evaluation#

Most existing approaches to evaluating text-to-3D scene generation rely on coarse similarity metrics that fail to capture whether objects are placed correctly relative to one another. They also overlook the fine-grained spatial reasoning needed to verify that, say, "apples on the table" means the apples are on top of the table, not merely present somewhere in the scene. This mismatch between how well systems seem to perform and what they actually deliver has made it difficult to drive real progress in the field.

Introducing LEGO and LEGO-Eval#

The new benchmark, called LEGO, pairs human-written descriptions with carefully annotated constraints and reference 3D scenes. Alongside it comes LEGO-Eval, an evaluation framework that breaks descriptions into atomic constraints—precise, individual requirements—and systematically verifies each one. The framework uses tools to ground language references to specific 3D objects and reason about their spatial relationships, such as containment, adjacency, or support.

Current systems fall well short#

Testing existing scene synthesis approaches against this stricter standard reveals sobering results: even the best current methods achieve only around 10% success rate on LEGO-Eval. This stands in sharp contrast to how much better these same systems appeared to perform under weaker evaluation metrics, suggesting the field has been underestimating the actual difficulty of the task.

The work highlights both the value of fine-grained constraint verification and the substantial engineering challenges that remain before text-guided 3D scene synthesis can reliably meet user expectations.