AI
Airbnb's CASTLE Framework Trains Search AI Using Synthetic Data From Real Bookings
A new technique generates training data for natural language search models by extracting booking patterns, reducing data quality gaps that plague early-stage deployments.
InfoQ
Airbnb has unveiled CASTLE, a machine learning framework that tackles a persistent obstacle in natural language search systems: training effective models when real user query data is scarce. Rather than waiting for guests to generate sufficient search traffic, the approach synthesizes training examples by analyzing actual booking behavior and leveraging large language models to create both realistic queries and relevance assessments.
The Core Challenge#
Platforms like Airbnb want guests to search using natural language descriptions—imagine asking for a "romantic cabin near skiing" or a "pet-friendly beach house for eight"—rather than manually selecting from fixed categories. Building AI models for this task requires two critical inputs: plausible search queries that mirror how real users think, and labels showing which property listings actually match those queries. Getting both right before real search volume arrives is exceptionally difficult.
How CASTLE Works#
The framework takes an elegant approach: it examines de-identified booking sessions to find contrastive listing pairs. When a guest books one property after viewing others, the booked property becomes a positive example and an unbooked alternative becomes a negative example. An LLM is then instructed to generate a query that fits the positive listing but would not match the negative one. This design embeds relevance labeling into the generation process itself, eliminating the need for a separate evaluation step and reducing error compounding that plagued earlier methods like InPars and Promptagator.
The team combined three query-generation strategies:
- Template-based prompts that follow structured patterns
- Few-shot examples using roughly 500 seed queries from user research
- Attribute-grounded prompts highlighting what distinguishes one listing from another
A built-in variation mechanism ensures the synthetic queries span different lengths and phrasings, mimicking real diversity.
For newly listed properties or listings with thin booking histories, CASTLE deploys a Virtual Judge—another LLM-based component that assigns relevance labels to additional query-listing pairs. A production variant, CASTLE-R, incorporates real search traffic as it accumulates.

