AI
LinkedIn Cuts LLM Ranking Latency 10x by Compressing Job Descriptions into Single Embeddings
Through model distillation and aggressive context compression, LinkedIn reduced inference latency for LLM-based search ranking while maintaining quality—and discovered that dense vectors outperform summarized text.
InfoQ
LinkedIn's search and recommendation systems now serve 1.3 billion members across natural-language queries, making LLM-based ranking at scale both compelling and fiendishly complex. At a recent InfoQ presentation, tech lead Sundara Raman Ramachandran detailed how the platform engineered its way past strict latency budgets and massive concurrency to deploy LLMs for real-time ranking decisions—from job searches to network connections.
The Ranking Problem Differs From Text Generation#
LLM ranking is fundamentally different from generative workloads. While a chatbot prefill-then-decode architecture scales with output tokens, ranking requires scoring dozens or hundreds of candidates against a single query. LinkedIn's approach creates 50+ prompts per request (one per job), each sharing a long common prefix (the query and system prompt), but needing only the final yes/no token probabilities—no token generation loop. This architecture demands brutal efficiency: P99 latency SLAs in seconds, not minutes, and hundreds of concurrent ranking requests.
Model: Compress Through Distillation#
LinkedIn started with a 7-billion-parameter teacher model capable of reasoning about relevance and engagement. Rather than deploy it directly, they distilled its knowledge into two smaller 0.6B models—one optimized for relevance, one for engagement—then further compressed both via multi-teacher distillation into a single 0.6B student model. The team then applied pruning, removing layers and attention heads, and re-distilled iteratively, arriving at a 400M production ranker with acceptable quality loss.
Context: From 2,000 to 1 Token#
A job description averages 2,000 tokens; ranking 100 candidates meant processing 200K tokens per request—infeasible at LinkedIn scale. The team pursued two sequential optimizations:
- Summarization: An offline LLM reduced job descriptions from 2,000 to 150 tokens (16× compression) with less than 2% quality loss. Throughput jumped from 290 to 2,200 items/second/GPU.
- Dense embeddings: The breakthrough came from compressing full job descriptions into a single dense vector. An embedding model converted the entire description offline into one token-equivalent representation. Throughput leapt to 22,000 items/second—a 10× gain over summarization—and versus the 150-token summaries.

