Databases
Google Brings LLM Inference Inside AlloyDB with Proxy Models That Cut Costs 6,000x
AlloyDB now runs large language models directly within SQL queries using local proxy models trained on sample data, enabling per-row AI operations at database speed.
InfoQ
Google has rolled out general availability for AlloyDB AI functions alongside two new acceleration techniques that reshape how databases interact with large language models. The approach moves LLM execution from external API calls into the database itself, radically lowering latency and cost.
The Challenge at Scale#
Processing large datasets with per-row LLM decisions creates a familiar bottleneck: a table of 100,000 products demands 100,000 separate API calls to Vertex AI. Each call transmits the same system prompt, waits for model inference, and incurs per-token charges. This row-at-a-time pattern becomes prohibitively expensive and slow as datasets grow.
Two Acceleration Paths#
Smart batching groups multiple rows into a single model invocation. Rather than repeating the system prompt for each row, AlloyDB bundles data and sends it once. Google's internal testing shows this delivers throughput up to 10,000 rows per second—a 2,400x gain over the baseline. The improvement is credible for workloads where the per-row payload remains small relative to the prompt size.
Proxy models represent the more fundamental shift. For ai.if semantic filtering queries (currently in preview), AlloyDB introduces a two-phase workflow:
- A PREPARE statement samples your data and sends it to a frontier LLM, using the results to train a lightweight local model inside the database
- An EXECUTE phase runs the query using this local proxy instead of calling the external model—achieving 100,000 rows per second throughput
The database falls back to the external LLM if the proxy's confidence drops below a threshold or no trained model exists.
A Inverted Architecture#
This pattern reverses the traditional database-to-LLM relationship. Instead of the database acting as a client polling an external model for every decision, it becomes a student learning the model's judgment on a sample, then applying that judgment locally at full database speed. The LLM shifts from runtime dependency to teacher.
What's Available and What's Coming#
AlloyDB's GA release includes ai.generate, ai.if, , , , , and . Smart batching is production-ready for and ; the optimized proxy model remains in preview.
