AI
New taxonomy maps 73 types of runtime failures in AI tool integration servers
Researchers identify widespread reliability issues in Model Context Protocol servers that connect large language models to external tools and data.
A new study offers the first systematic classification of runtime faults affecting Model Context Protocol (MCP) servers—the infrastructure that allows large language models to interact with external tools and data sources. The research reveals that these systems face far more complex failure modes than previously documented.
What the study examined#
Researchers conducted an empirical investigation by analyzing 833 confirmed runtime fault reports extracted from 473 actively maintained MCP server repositories on GitHub. Using an open-coding methodology, they built a bottom-up taxonomy that captures the full spectrum of failures developers encounter in practice. The analysis identified 11 major fault categories broken down into 27 subcategories, ultimately covering 73 distinct fault types.
Types of faults identified#
The failures span several operational areas:
- Protocol interactions — breakdowns in how MCP servers communicate with language models
- Tool invocations — problems when executing external functions or commands
- Schema enforcement — violations of expected data structure specifications
- State management — errors in maintaining consistent internal configurations
- Model-provider integration — issues connecting to external AI services
- Security validation — flaws in access control and data protection
- Operation timing — timeouts or improper cancellation of in-progress requests
A recurring theme involves configuration parameters that systems accept during setup but fail to enforce during runtime, causing unexpected default behaviors and cascading failures.
Validation through developer feedback#
To verify the taxonomy's real-world relevance, the researchers surveyed 55 MCP server developers. Respondents reported experiencing an average of 20 out of 27 fault subcategories, with no category remaining completely unobserved. This widespread agreement suggests the taxonomy captures the actual challenges facing production systems.
