Cutting GPU Model Startup Time from 8 Minutes to Under 1 Minute: Six Layers, Six Fixes

The Problem: Eight Minutes of Wasted GPU Time#

Running large language models on GPUs involves far more than just loading weights and generating tokens. When a new inference pod starts on a freshly provisioned GPU node, it must pass through six distinct phases before serving its first request—and those phases can consume as much as 8–15 minutes on a cold start. That's expensive GPU compute time burning away while nothing useful is happening.

Researchers instrumented the full path from pod creation to first inference response and discovered not one bottleneck, but six. More importantly, which bottleneck dominates depends entirely on model size: for a 64 GB model, 65% of startup time is spent recompiling CUDA kernels that produce identical output every time. For a 203 GB model, 92% vanishes into S3 weight downloads that fail to saturate available bandwidth. Both issues are fixable through configuration changes. Neither is fixed by default.

The Six Layers of Cold Start#

The startup sequence breaks down into six distinct phases, each with its own bottleneck and its own solution:

Layer 1: Node provisioning (~60–90s) — Karpenter launches an EC2 instance, boots it, and registers it with Kubernetes. This is hardware floor; no optimization removes it entirely.

Layer 2: GPU driver initialization (2–3 minutes) — NVIDIA's GPU Operator compiles the driver kernel module from source on every new node. Pre-compiling this at image build time—enabled by platform-level control of the full stack—cuts this to seconds.

Layer 3: Container image pull (2–4 minutes) — Inference engine images are 8–12 GB compressed. Standard sequential pull, decompression, and disk write takes minutes. SOCI's parallel snapshotter downloads chunks concurrently via HTTP range requests and decompresses across all available CPU cores, shrinking this to 30–60 seconds on well-provisioned instances.

Layer 4: Model weights download (25–423 seconds depending on model size) — Naive approaches like spawning 256 parallel connections don't work because the underlying calling pattern processes range requests sequentially within each worker thread. Aligning chunk size to shard file boundaries and using aggressive timeout-and-retry for stalled S3 connections reduces a 203 GB model's load from 423 seconds to 25 seconds—a 94% improvement with no code changes.

Layer 5: GPU kernel compilation (34–53 seconds) — PyTorch traces the model's computation graph and generates optimized CUDA kernels every time a pod starts, even though the output is identical. Pointing the torch.compile cache directory at local NVMe instance store—which EKS Auto Mode mounts automatically—cuts this to 4–6 seconds on the second pod running on the same node.