AI
MIRAGE: LLMs Simulate Microservice Dependencies in Real-Time for Better Testing
Researchers propose using large language models to dynamically simulate microservice behaviors during testing, achieving far higher accuracy than static mock methods.
A team of computer scientists has developed a novel approach to testing microservice systems by having large language models generate realistic dependency simulations on demand, rather than relying on pre-recorded artifacts.
The Problem with Static Mocking#
Conventional microservice testing relies on generating static mocks before tests run—through recording, pattern mining, or specification-driven stubs. These approach has a critical weakness: they can only reproduce behaviors that were captured at generation time. This limitation becomes painfully apparent when testing error handling or complex code-reasoning scenarios, which are typically underrepresented in production trace collections. In the researchers' evaluation, traditional record-replay methods achieved 0% fidelity on error-handling tests and just 12% on code-reasoning scenarios.
How MIRAGE Works#
The proposed solution, called MIRAGE, takes a fundamentally different approach. Instead of generating static artifacts upfront, it uses a large language model to process each incoming dependency request as it arrives during test execution. The model has access to three key inputs: the dependency's source code, the calling code, and historical traces from production systems. This allows it to reason through appropriate responses on the fly, maintaining consistent state across multiple requests within a single test scenario.
The tradeoff is real—each simulated request takes roughly 3 seconds and costs between $0.16 and $0.82 per dependency call. However, this runtime inference enables the system to handle scenarios that static mocks completely miss.
Evaluation Results#
The researchers tested MIRAGE across 110 test scenarios spanning three microservice systems: Google's Online Boutique, Weaveworks' Sock Shop, and a custom test system. The results were striking:
- White-box mode (with full source code access): MIRAGE achieved 99% fidelity on status codes and 99% on response structure, compared to 62% and 16% respectively for traditional record-replay.
- Code importance: Dependency source code alone proved sufficient for high accuracy (100% correct), while removing it still maintained 94% error-code accuracy but dropped response-structure fidelity to 75%.
