LLMs Show Promise for Tax Law Reasoning, But Aren't Ready for Full Automation

Researchers have examined whether large language models can reliably assist tax consultants in analyzing Austrian and EU value-added tax cases. The work, published on arXiv, reveals that while LLMs can provide meaningful support for routine tax analysis, current systems fall short of being trustworthy enough for independent legal decision-making.

The Challenge of Automating Legal Reasoning#

Tax consulting often begins with clients describing their situations in plain language. That format makes LLMs seem like an ideal tool for automating initial case assessment and reducing the administrative burden on professionals. However, the legal domain presents a unique obstacle: LLMs' tendency to fabricate plausible-sounding information—known as hallucination—poses serious risks when accuracy and justified reasoning are essential.

Testing Two Enhancement Methods#

To improve LLM performance on tax law tasks, researchers experimented with two established techniques. Fine-tuning involved training models on domain-specific examples, while retrieval-augmented generation (RAG) supplemented the models with relevant documents at inference time. Both approaches were evaluated using textbook cases and real-world scenarios from a tax consulting practice, allowing the team to identify the most effective configurations for each scenario.

Mixed Results on Real-World Application#

The findings suggest that properly configured LLM systems can generate legally sound justifications and support tax professionals in their work. However, significant limitations persist. The models struggled with implicit client context and specialized documentation that required deeper background knowledge to interpret correctly. These gaps mean human oversight remains essential, particularly for complex or novel cases.

Path Forward#

The research underscores LLMs' potential to automate routine tasks and speed up preliminary analysis in tax consulting. Yet the legal sensitivity of the domain and current model limitations make them unsuitable for full automation today. Integrating structured background information and improving the handling of context-specific details are among the priorities for future development.