Serverless GPU inference allows teams to pay only for the tokens they generate. For startups scaling from zero to millions of users, the ability to scale-to-zero during idle periods is a financial necessity. However, the technical reality often involves a catastrophic latency penalty. When a request hits an idle endpoint, the infrastructure must provision a GPU, pull a multi-gigabyte container image, load weights into VRAM, and initialize the inference engine. In a production environment where users expect sub-second responses, a 40-second cold start is a failure. Engineering teams must move beyond basic auto-scaling and address the underlying bottlenecks of the GPU stack.
Inference Serving Cold Starts 7 min read
Serverless Inference Cold Start Latency: A Technical Optimization Guide
Cold starts remain the primary barrier to responsive serverless AI. This guide breaks down the technical stages of GPU initialization and provides a framework for minimizing latency in production environments.
Magnus Grünewald
April 23, 2026 · CEO at Lyceum Technology
Last updated August 3, 2026
The Four Stages of GPU Initialization
To solve cold start latency, you must first understand where the time is spent. A typical GPU cold start is not a single event but a sequence of four distinct phases, and the split between them depends on the engine, the model and where the weights are staged. The first systematic characterization of vLLM startup, published in June 2026, breaks its start into six steps and finds it predominantly CPU-bound [3].
- Infrastructure Provisioning: The cloud orchestrator identifies an available GPU and assigns it to your workload. Google documents Cloud Run instances with an attached L4 GPU and drivers pre-installed starting in approximately 5 seconds [1]. Where capacity is constrained, it can instead stall for minutes.
- Container Image Pulling: AI containers are heavy, often exceeding 10GB once CUDA libraries and framework dependencies are included. Google puts the practical ceiling for baking a model into an image at 10GB, warns that the build bottlenecks on network throughput, and recommends downloading weights from object storage instead [2].
- Model Weight Loading: Transferring weights from disk or network storage into GPU VRAM is limited by the PCIe bus or by network bandwidth. A Llama 3 70B model in FP16 carries roughly 140GB of weights, so where those bytes are staged decides how long this takes [2].
- Engine & CUDA Context Setup: The inference engine (such as vLLM or TensorRT-LLM) must initialize the CUDA context, allocate the KV cache, and capture CUDA graphs. Measured on vLLM, this phase is predominantly CPU-bound, which is why a faster accelerator does not shorten it [3].
The total latency is the sum of these parts. A warm replica skips the cold start and answers in milliseconds; under scale-to-zero, the first request after an idle period pays the whole sequence. That gap is what renders naive scale-to-zero architectures unusable for real-time applications like voice AI or interactive coding assistants.
The Scale-to-Zero Paradox: A Decision Framework
Choosing between serverless and dedicated infrastructure is a trade-off between idle costs and user experience. If your application is latency-sensitive, scaling to zero might be a false economy. Conversely, for batch processing or internal tools, paying for a 24/7 H100 instance is wasteful. Use the following framework to determine your deployment strategy.
| Workload Type | Latency Tolerance | Recommended Model | Cost Driver |
|---|---|---|---|
| Interactive Chat / Voice | < 500ms | Dedicated / Warm Pool | Uptime |
| Code Completion | < 200ms | Dedicated | Uptime |
| Batch OCR / Parsing | > 30s | Serverless | Per-token / Per-job |
| Medical Image Segment. | < 2s | Warm Pool / Fast Serverless | Hybrid |
Common Deployment Mistakes
Many teams attempt to use "keep-alive" pings to prevent scale-to-zero. While this works for simple Lambda functions, it is inefficient for GPUs. A single H100 instance can be prohibitively expensive when left idle. If you are pinging it every 5 minutes to keep it warm, you are effectively paying for dedicated infrastructure but without the reliability of a reserved instance.At Lyceum, we address this by provisioning VMs and clusters from pre-staged images and capacity rather than building an environment on every request. That shortens the infrastructure phase, letting teams stay in the serverless execution model longer before the latency penalty forces a move to dedicated hardware.
Modern Orchestration: NVIDIA Dynamo
The release of NVIDIA Dynamo has fundamentally changed how we manage inference at scale. Positioned as a distributed "operating system" for AI factories, Dynamo introduces several features that directly mitigate cold start issues.
- KV Block Manager (KVBM): Offloads the Key-Value cache across GPU, CPU, SSD and remote storage, extending effective context length beyond GPU memory and avoiding a full cache rebuild on every start [4].
- NIXL (NVIDIA Inference Xfer Library): High-speed GPU-to-GPU data movement, so a warm node can hand state to a newly provisioned one. NVIDIA's ModelExpress streams model weights over NIXL and NVLink, and the Dynamo repository reports 7x faster model startup for DeepSeek-V3 on H200 [4].
- KV-Aware Routing: Dynamo routes requests by worker load and KV cache overlap, so a request can land on a GPU that already holds the relevant prefix and skip redundant prefill work [4].
By integrating Dynamo with open-source engines like vLLM, infrastructure providers can cut re-initialization work and raise throughput on Blackwell-class GPUs. This orchestration layer sits above the raw hardware, acting as a traffic controller that minimizes the need for full re-initialization. For European teams, using an open-stack implementation of Dynamo ensures portability, avoiding the black-box lock-in common with US-based API providers.
Modern Technical Strategies for GPU Inference
If you are building a production-grade inference stack, standard optimization is no longer enough. You must implement advanced techniques to bypass the physical limits of model loading.
Model Weight Streaming
Rather than waiting for the entire 140GB model to load, modern runtimes use lazy loading or weight streaming. The engine begins generating the first token as soon as the first few layers are in VRAM. This significantly reduces Time to First Token (TTFT), even if the total load time remains the same.Filesystem Snapshotting
Technologies like CRIU (Checkpoint/Restore in Userspace) allow you to save the entire state of a running container, including the initialized CUDA context and loaded weights. Restoring from a snapshot is often much faster than a fresh start because it bypasses the framework initialization and graph capture phases.VRAM Prediction
Lyceum's scheduling product uses runtime estimation and VRAM prediction to select a suitable GPU for a specific job. By predicting the memory requirements of a request before it hits the hardware, the scheduler can place work on a node that already has the base model cached, which cuts wasted allocation and reduces latency. Memory and runtime prediction within a node is shipped; multi-node prediction beyond 4 GPUs per node is in development.Batch Processing Scenario
A document parsing startup needs to process 10,000 PDFs in a batch. Using standard serverless, each job might trigger a cold start. By using a warm pool with predictive scaling, the startup can process the entire batch with only a single initial cold start, then scale back to zero once the queue is empty.The Sovereignty Moat: Why Location Matters for Latency
For European AI teams, latency is not just a hardware problem; it is a geographic one. When you use US-based inference providers, every request must cross the Atlantic. Microsoft's published Azure medians, for the 30 days ending 30 July 2026, put the European hop to the US East Coast at 73 to 100 milliseconds [5]. For real-time applications, this "latency tax" is often the difference between a fluid user experience and a clunky one.
Furthermore, GDPR transfer rules make the location of processing a hard requirement for many regulated buyers, and the EU AI Act adds documentation duties on top. Moving data to US servers for inference is a deal-breaker for regulated industries like healthcare, defense, and manufacturing. EU-hosted inference platforms keep processing inside European data centres. By hosting your models in Paris or the Nordics, you remove the trans-Atlantic hop and the GDPR Chapter V transfer question.
Serving inference from European data centres in Spain, Paris and the Nordics also simplifies the commercial side: GPU compute is billed per second with no base fee. S3-compatible storage carries no ingress or egress charge, and current GPU-hour and per-token rates are published on the pricing page rather than quoted per deal. Combine that with the performance gain of local data residency and the trade-off for European scale-ups is easy to evaluate.
Sources
[1] Google Cloud: GPU support for services, Cloud Run, read 3 Aug 2026; [2] Google Cloud: Best practices, AI inference on Cloud Run with GPUs, read 3 Aug 2026; [3] Breaking the Ice: Analyzing Cold Start Latency in vLLM, arXiv:2606.07362, June 2026; [4] NVIDIA: Dynamo inference stack, repository README, read 3 Aug 2026; [5] Microsoft Azure: Network round-trip latency statistics, P50 medians for the 30 days ending 30 July 2026, read 3 Aug 2026
Frequently Asked Questions
Why are AI containers so large?
AI containers bundle heavy dependencies like CUDA, cuDNN, and PyTorch, along with the model weights themselves. A standard environment can easily exceed 10GB, which is why container pulling is a major cold start bottleneck.
How does Lyceum handle cold starts?
Lyceum's scheduling product uses VRAM prediction, and VMs are provisioned from pre-staged capacity, shortening the infrastructure phase. A warm replica skips the cold start; with scale-to-zero, the first request after idle pays one.
Can I use the OpenAI SDK with Lyceum?
Yes. Lyceum Inference Studio is OpenAI-compatible. You point your code at the base URL shown in your Lyceum dashboard.
What is the impact of EU data residency on latency?
Hosting in the EU removes a trans-Atlantic hop that Microsoft's Azure tables put at 73-100 ms. European users get a more responsive application than a US-hosted service.
Is per-second billing available for all GPUs?
Yes. Lyceum bills GPU compute per second across its GPU types, including H100, A100 and B200, with no subscription or base fee. Reserved capacity has a minimum of one month and one server.
Lyceum Technology