Kubernetes struggles to track true AI inference costs as token economics reshape cloud-native workloads

Kubernetes is increasingly becoming the operational backbone for AI inference in production environments, with major enterprises deploying thousands of accelerators for model serving. Yet a fundamental mismatch persists: the orchestrator's traditional resource accounting—designed for generic compute workloads—may not align with the token-based economics that now define large-scale AI inference costs.

The economics shift from compute to tokens#

The tension emerged this week as the ecosystem prepares for a dedicated AI Inference + Agentic track at KubeCon + CloudNativeCon North America 2026. Val Bercovici, chief AI officer at WEKA, highlighted the core problem: pricing per token depends on variables Kubernetes was never built to track—specifically request patterns, KV cache occupancy, the ratio of prefill to decode operations, and how memory and bandwidth are actually consumed within accelerators once a pod is running.

"My view is that Kubernetes doesn't go away. But unless its resource model evolves, it becomes a tax on inference economics," Bercovici told The New Stack. He predicts that new scheduling and memory layers will need to emerge atop Kubernetes to compute real token-serving costs, enabling platforms to make genuinely cost-based decisions about workload distribution.

Real-world wins point to what's possible#

Despite these challenges, practical deployments show meaningful progress. China Merchants Bank, a major Chinese financial institution, recently won the CNCF End User Case Study Contest with a cloud-native AI infrastructure combining Kubernetes and open-source projects including Kueue, KEDA, Prometheus, HAMi, and Fluid. The bank manages a heterogeneous pool of nearly 10,000 accelerator cards for AI workloads.

Results demonstrated what focused optimization can achieve:

• Unified management of 99% of AI compute resources
• Lifted average utilization from 35% to over 60%
• Reduced the cost per million tokens by 60% under comparable conditions

The case illustrates that Kubernetes can handle sophisticated, large-scale inference scenarios even in regulated industries—but success requires integrating multiple specialized tools to address gaps in native resource accounting.