Study clarifies how language models function in open-ended military simulations

A new scoping review from arXiv examines how language models are deployed in wargaming and strategic simulations, highlighting a critical but often overlooked distinction: whether models control what actors do, what happens as a result, or both.

The control-profile problem#

When researchers rely on transcripts generated by language models in simulations, the output can seem like a unified behavioral snapshot. Yet beneath the surface, the underlying model may be doing fundamentally different tasks. In some cases, it chooses character actions alone, in others it adjudicates outcomes—what the researchers call determining "what happens after an action"—and in the most capable setups, it handles both roles simultaneously.

This distinction becomes especially important in open-ended wargames, where handling novel, unexpected actions and ambiguous consequences is a central requirement. A model that merely selects dialogue has vastly different constraints and capabilities than one that also interprets the ramifications of those choices.

Key findings from 223 studies#

The review analyzed 223 deduplicated papers on AI applications in wargames and strategic simulations, cataloging each by its model-control profile. The results were stark: fewer than one in ten studies (~9%) assigned language models responsibility for both player actions and adjudication simultaneously. The overwhelming majority limited models to a single function.

This fragmentation suggests that most research has sidestepped the harder problem of giving models both creative agency and interpretive authority in the same simulation.

Why fidelity matters#

For truly open-ended simulations to be credible, fidelity must rest on two pillars: plausible agent behavior and reliable adjudication. A simulation where actors speak convincingly but an automated referee makes nonsensical rulings about consequences is no more authentic than one where outcomes are sensible but dialogue feels robotic.

Before drawing conclusions from language-model-based social simulations, researchers need transparency about the scope of each model's control—a prerequisite for understanding whether the transcript reflects genuine simulation dynamics or an incomplete artifact of constrained automation.