AI
Uber's AI-powered chaos testing finds critical mobile app failures at scale
A machine learning system that automatically tests mobile apps against backend failures has identified dozens of hidden vulnerabilities in Uber's Rider, Driver, and Eats platforms.
Researchers from Uber have developed an automated system that uses large language models to execute chaos tests on mobile applications at unprecedented scale, uncovering resilience issues that traditional testing cannot detect.
The Challenge at Scale#
Mobile applications deployed across global infrastructure face constant risk from backend service failures. However, conventional chaos engineering struggles with mobile testing because the number of possible combinations—different user workflows, geographic locations, and fault scenarios—grows exponentially. Manually writing test cases for each combination is impractical for large-scale systems.
AI-Driven Automated Testing#
The research team integrated DragonCrawl, an LLM-based mobile testing platform, with uHavoc, a service-level fault injection system. Rather than hardcoding specific test scenarios, the AI-driven approach learns to navigate through mobile applications while simulating degraded backend conditions, automatically discovering whether the apps behave correctly under various failure modes.
Since early 2024, the system has executed more than 180,000 automated chaos tests across 47 critical user flows in Uber's Rider, Driver, and Eats applications. This scale of testing would require approximately 39,000 hours of manual work—clearly unscalable with human testers alone.
Key Findings#
The testing identified 23 resilience risks. Notably:
- 70% of issues stemmed from architectural dependency violations where failures in non-critical backend services cascaded to break core user experiences
- 12 problems were severe enough to prevent trip requests or food order placement
- 2 issues caused application crashes that appeared only when testing mobile apps—backend testing alone would have missed them
The system's automated root cause analysis pinpointed which backend service caused each mobile failure with 88% precision, cutting debugging time from hours to just minutes. Under fault injection testing, the system maintained 99% reliability, confirming it works correctly even when the applications themselves are failing.
