AI
Cloudflare's AI Search Helps Agents and Developers Index and Query Custom Data
The platform now bundles search infrastructure into a unified service that automates crawling, embedding, and retrieval for AI applications.
InfoQ
Cloudflare has extended its AI Search offering, a managed search and retrieval system designed to give AI agents and applications direct access to custom datasets. By combining multiple infrastructure components into a single pipeline, the service eliminates the need for developers to assemble and maintain separate tools for crawling, parsing, vectorizing, and querying data.
What AI Search Does#
Rather than forcing developers to wire together individual components—a crawler, text parser, embedding model, vector database, and search API—Cloudflare's service handles this stack automatically. The company draws on existing primitives it already offers: Workers AI for compute, Vectorize for vector storage, R2 for object storage, Browser Run for web interaction, and others. AI Search orchestrates these into an end-to-end search pipeline that agents can query for context.
Organizations can now index both structured and unstructured data and expose it as a searchable corpus. Previously, indexed websites had to publish sitemaps for discovery; the updated service operates in a "discover" mode that automatically surfaces pages without requiring XML manifests.
Easier Setup and Integration#
A search instance requires just a single command:
npx wrangler ai-search create cloudflare-community \
--namespace dev-stack \
--source https://community.cloudflare.com \
--type web-crawler \
--parse-type discover
Cloudflare provides a public endpoint that allows searches across multiple websites or instances simultaneously without authentication. For users running EmDash, an open-source CMS, there's a dedicated plugin for straightforward integration.
The service already indexes Cloudflare's own API documentation, developer guides, and third-party frameworks including Astro, Vite, Hono, and Replicate. This unified corpus allows a single query to pull context from multiple sources at once—useful for agents that need breadth across related documentation.
