Shopify Shrinks LLM Prompts into Learned Tokens, Cutting Latency and Costs

Shopify has unveiled Gisting, a method that distills lengthy language model prompts into a compact set of learned tokens, delivering measurable gains in speed, cost, and inference throughput without altering the model itself.

How Gisting Works#

Rather than storing or processing full text prompts, Gisting compresses them into special "gist" tokens—learned embeddings that encode the essence of the original instructions. The approach uses a two-pass training process:

• Teacher pass: The model runs with the full, uncompressed prompt to generate reference output probabilities (teacher logits)
• Student pass: The model processes the compressed gist tokens and produces its own output probabilities (student logits)

The gist tokens are then refined to minimize the statistical divergence between teacher and student outputs, ensuring the model behaves identically to receiving the original prompt. Once training completes, these embeddings are written directly into the model's token vocabulary—no custom serving logic or architectural changes required.

Real-World Performance Gains#

Shopify tested Gisting on its Sidekick GraphQL agent, compressing a 6,000-token system prompt into just 1,500 gist tokens—a 4:1 reduction—without degrading prediction quality. At 350 requests per minute, the improvements were substantial:

• Time to first token: fell from 438ms to 354ms
• End-to-end latency: dropped from 6.8 seconds to 4.2 seconds
• Throughput: increased from 20.2 to 23.4 queries per second

These gains allowed the team to scale back GPU allocation, directly cutting infrastructure costs.

Complementary with Existing Optimizations#

Gisting works alongside other inference techniques, notably prefix caching, which reuses precomputed key-value tensors for repeated prompt segments. While prefix caching saves computation, the model still must process those cached values during generation. Gisting reduces this further by replacing the prompt entirely with a shorter token sequence, allowing both methods to compound their benefits when used together.

Read the full technical writeup on Shopify's engineering blog.