Foundation models falter on engineering: New benchmark reveals weak multimodal reasoning

Foundation models—the large neural networks powering modern AI systems—have impressed across many domains, yet their grasp of real-world engineering challenges remains unclear. A new study introduces SoM-1K, an extensive multimodal benchmark designed to stress-test these models on strength of materials problems, a core discipline in mechanical and civil engineering.

The benchmark#

The dataset contains over 1,000 annotated problems that combine textual descriptions and schematic diagrams, mirroring how engineers encounter problems in practice. Researchers evaluated eight prominent foundation models, spanning both general-purpose large language models (LLMs) and models designed to handle both text and images (VLMs). The findings are sobering: even the best-performing system achieved only 56.6% accuracy, a marked gap from human-level performance.

A novel workaround#

Recognizing that current AI systems struggle to parse complex technical diagrams, the researchers developed Descriptions of Images (DoI)—a method in which human experts generate precise text summaries of visual schematics. When LLMs received these text descriptions alongside problem statements, they often outperformed VLMs given the raw diagrams. This counterintuitive result points to a fundamental weakness: direct image interpretation, despite being theoretically more complete, proved less reliable than carefully written textual proxies.

Why it matters#

The findings highlight critical blind spots in how foundation models process multimodal information, especially in domains requiring rigorous technical reasoning. The work signals an urgent need to build AI systems more attuned to scientific and engineering contexts, where precise visual understanding and logical inference are intertwined. Accurate text-based descriptions, it seems, can compensate where direct vision-based analysis fails—at least for now.