Cloud Infrastructure
Amazon EKS shrinks multi-gigabyte ML image pulls from minutes to seconds
Machine learning workloads with 20–30 GB container images can now pull in seconds instead of minutes on Amazon EKS by rethinking the download and unpacking pipeline.
The New Stack
Large machine learning container images—often 20–30 GB or bigger—have become a significant bottleneck in production Kubernetes clusters. While traditional applications ship in hundreds of megabytes, modern ML inference stacks bundle deep-learning frameworks, CUDA libraries, and model weights, creating images that take several minutes to pull onto GPU-accelerated instances. During that time, expensive accelerators sit idle waiting for the image to arrive.
The real constraint was software, not hardware#
A production team running ML workloads on Amazon EKS faced exactly this problem: they needed pods ready in two minutes, but pulling a 30 GB image alone consumed most of that window. Profiling revealed the bottleneck wasn't the network—the instances had 100–400 Gbps of available bandwidth—but rather how the container runtime used the resources already at hand.
The container image pull process normally unfolds across six sequential stages: downloading compressed data, verifying its cryptographic hash, writing it to disk, decompressing it, hashing the uncompressed content, and extracting files into the filesystem layer. Each of these operations traditionally happened one layer at a time, leaving compute, network, and disk throughput underutilized as the system waited for each stage to finish.
Parallel downloads and concurrent unpacking#
The team identified two major inefficiencies and addressed them:
-
Chunked parallel downloads: Instead of fetching each layer over a single HTTP connection, they split large layers into fixed-size chunks and downloaded them concurrently using HTTP range requests. This meant a layer that previously took over a minute arrived in seconds, while keeping memory usage constant by writing chunks directly to disk rather than buffering them in RAM—a critical advantage on GPU nodes where system memory is shared with model weights.
-
Concurrent layer unpacking: The traditional sequential unpacking (layer two can't start until layer one finishes all six stages) was replaced with parallel decompression and extraction. Because the default overlay filesystem backend stores each layer separately and only stacks them at runtime, there's no ordering dependency. A 30 GB image that previously required summing the unpacking time of all layers now finishes in roughly the time of the largest single layer.
