LLMs Show Promise as Choice Modelling Assistants, But Performance Varies Sharply

Large language models are increasingly being deployed across professional and research workflows, but their utility for econometric tasks like discrete choice modelling—a method economists use to understand how people make decisions between alternatives—remains largely untested. A comprehensive new study benchmarks how well leading LLMs can assist with building multinomial logit (MNL) models, a core tool in choice analysis.

Study Design and Scope#

Researchers tested twelve versions of seven major LLMs—ChatGPT, Claude, DeepSeek, Gemini, Gemma, Llama, and Mistral—across five experimental setups. The models were evaluated on their ability to suggest utility function specifications for choice models, and in some cases, to estimate them directly. Three key variables shaped the experiments:

• Modelling goal: whether the LLM simply suggested model specifications or was also asked to estimate them
• Prompting approach: zero-shot prompts versus Chain-of-Thought (CoT) structured reasoning
• Information access: exposure to full datasets versus summaries listing variable names and types

Each suggested specification was then implemented, fitted to data, and assessed using standard metrics like goodness-of-fit, behavioural validity, and model complexity.

Key Findings#

The results split sharply along model type. Proprietary LLMs—particularly those using structured Chain-of-Thought prompting—generated valid and theoretically sound utility specifications. Interestingly, models sometimes performed better when given only a data dictionary rather than raw data, suggesting that less information can focus the model's reasoning more effectively.

Open-weight alternatives like Llama and Gemma struggled to produce meaningful specifications, highlighting a current performance gap between commercial and community-developed models. A notable exception was GPT-o3 operating in an agentic mode, which uniquely succeeded at writing and executing its own code to estimate its generated specifications—effectively closing the loop on the full modelling pipeline.

Implications for Practitioners#

The findings suggest LLMs can meaningfully support choice modellers in specification design and decision support, but with important caveats. Structured prompting and careful thought about what information to share with the model can improve outcomes. The work provides practical guidance for incorporating these tools into econometric workflows while remaining realistic about their current limitations.