DoorDash Rebuilds AI Safety System as Reusable Moderation Platform

DoorDash encountered a real-world problem at scale: the platform exchanges over 4 million chat messages daily between consumers, delivery drivers (Dashers), and restaurants, with hundreds of thousands of voice calls and 200,000+ images flowing through the system each day. A meaningful share of safety incidents involved verbal abuse—and because these marketplace relationships last minutes to an hour at most, the company needed to act instantly when something unsafe was detected.

The Initial Challenge: Speed vs. Cost#

The naive approach—simply querying a large language model to classify each message as safe or unsafe—fell apart at DoorDash's scale. LLMs typically take 2 to 10 seconds per inference, and running 4 million calls daily would have generated prohibitive costs. Instead, Bruna Pereira's trust and safety engineering team at DoorDash took time to understand the actual problem before building a solution.

After analyzing millions of messages, the team discovered a crucial insight: only a small single-digit percentage of messages were actually unsafe. This finding shaped everything that followed.

SafeChat: A Tiered Architecture#

The team built a cheap first layer—a fast, lightweight classifier trained on their own labeled data. This model had to be quick (under 100 milliseconds), cost-effective (no per-call fees, just infrastructure), and excellent at identifying messages that were obviously safe.

For the roughly 10% of messages the classifier couldn't confidently clear, the system escalated to an LLM. Critically, instead of asking yes-or-no questions, the team asked the model to score messages across multiple dimensions—how threatening, how profane, how sexual—rather than rendering a binary judgment.

This scoring approach proved crucial. Graduated severity levels meant different actions: censoring low-severity swearing while letting it through, blocking mid-severity insults entirely, and for high-severity threats, canceling the order and offering the affected party an exit without cost.

Results and Rethinking#

SafeChat cut verbal-abuse-related incidents by roughly 50%—a real reduction in human harm, not merely a model accuracy metric. But as the system gained traction, other DoorDash teams asked to use the same pattern for their own problems: moderating profile pictures, reviewing food comments, detecting fraud in chats and calls.

The team realized they had invented something more generalizable than SafeChat itself.