Latency is no longer just a metric for user experience; it is the primary driver of unit economics. As teams move from prototyping to production, the cost of serving a model often outweighs the cost of training it. For a 100-person AI startup, a 200ms delay in Time to First Token (TTFT) can be the difference between a product that feels like a fluid conversation and one that feels like a broken utility. Reducing latency requires a multi-layered approach that spans from the CUDA kernel level up to the orchestration layer. We see many teams struggle with the transition from hyperscaler credits to sustainable infrastructure, often because they are running unoptimized stacks that leave a large share of GPU performance on the table.
Inference Serving Throughput 5 min read
Reduce LLM Inference Latency on GPUs: A Technical Guide
High latency in LLM inference drives up compute costs and degrades user experience. This guide explores the hardware and software strategies required to minimize Time to First Token (TTFT) and maximize throughput on modern NVIDIA GPUs.
Magnus Grünewald
April 21, 2026 · CEO at Lyceum Technology
Last updated August 4, 2026
Understanding the Latency Hierarchy: TTFT vs. TPOT
Before optimizing, you must distinguish between the two primary latency metrics. Time to First Token (TTFT) measures how quickly the model starts responding, which is critical for interactive applications. Time Per Output Token (TPOT) measures the speed of subsequent tokens, determining the overall reading speed. The closest thing to an industry latency benchmark is MLPerf Inference, whose llama2-70b Server constraints are a p99 TTFT of 450 ms and a p99 TPOT of 40 ms in the Interactive category, and 2,000 ms / 200 ms in the Conversational category.
Latency bottlenecks typically fall into two categories: compute-bound and memory-bound. During the prefill phase (generating the first token), the GPU is often compute-bound as it processes the entire input prompt in parallel. During the decoding phase (generating subsequent tokens), the GPU becomes memory-bound because it must fetch model weights and the KV cache from HBM (High Bandwidth Memory) for every single token generated.
Prefill Phase
Highly parallel, benefits from raw TFLOPS.Decoding Phase
Sequential, benefits from high memory bandwidth (GB/s).KV Cache
Grows with context length, leading to Out-of-Memory (OOM) errors if not managed.
Software Optimization: vLLM, TensorRT-LLM, and NVIDIA Dynamo
The choice of inference engine is the most significant software decision you will make. Standard PyTorch implementations are insufficient for production. Modern engines like vLLM and NVIDIA TensorRT-LLM use a technique called PagedAttention to manage the KV cache. This prevents memory fragmentation and allows for much higher batch sizes, which indirectly reduces latency by increasing throughput.
NVIDIA introduced Dynamo in March 2025 and shipped version 1.0 in March 2026, providing a standardized orchestration layer that bridges the gap between raw compute and high-level APIs. An open-stack approach combining vLLM with NVIDIA Dynamo ensures customer portability. Unlike black-box proprietary engines, this stack allows you to maintain control over your model weights while achieving performance parity with specialized API providers. By using continuous batching, these engines process new requests immediately rather than waiting for an entire batch to finish, cutting average wait times substantially in high-traffic scenarios.
Quantization Strategies: Balancing Precision and Speed
Quantization reduces the bit-precision of model weights, which decreases the amount of data the GPU must move from memory to the processing cores. Moving from FP16 (16-bit) to FP8 (8-bit) halves the bytes an H100 must read per weight, which is equivalent to doubling effective memory bandwidth during decoding. NVIDIA's published 4x figure for FP8 on Hopper is GPT-J-6B on H100 FP8 measured against an A100 FP16 PyTorch eager-mode baseline rather than a clean FP8-versus-FP16 comparison; the only clean 2x is architectural peak throughput: NVIDIA publishes 3,958 FP8 and 1,979 FP16 Tensor Core TFLOPS for the H100 SXM with sparsity, which is 1,979 against 989 dense.
Common quantization methods include:
AWQ (Activation-aware Weight Quantization)
Protects the most important weights to maintain accuracy at 4-bit precision.FP8
The current standard for H100 and B200 GPUs, with a good balance of speed and precision.INT8
Older but reliable for previous-generation hardware like the A100.
For teams running 405B-class open weights such as Hermes-4-405B, quantization is not optional. At BF16 the weights alone need roughly 810 GB, more than the 640 GB an eight-way H100 node provides; FP8 brings them to about 405 GB and leaves headroom for the KV cache. Reducing the memory footprint also frees HBM for a larger KV cache, which raises both the context length and the number of concurrent requests a GPU can hold before it has to evict or queue work.
Architectural Tactics: Speculative Decoding
Speculative decoding is a powerful technique where a smaller, faster "draft" model predicts the next few tokens, which are then verified by the larger "target" model in a single forward pass. If the draft model is correct, you generate multiple tokens in the time it would usually take to generate one. Gains depend on the workload: the original speculative decoding paper measured 2x to 3x acceleration on T5-XXL with identical outputs and no change to the model's weights.
However, speculative decoding requires careful implementation. If the draft model's acceptance rate is low (below 50%), the overhead of verification can actually increase latency. We recommend using a draft model from the same family as your target model, for example, using a Llama 3.2 3B model to speculate for Llama-3.3-70B. This ensures a higher alignment in token distribution and better performance gains.
Infrastructure and Data Residency: The Hidden Latency
Network latency often negates GPU-level optimizations. For European startups, hosting models in US-based data centers adds 100ms to 150ms of round-trip time (RTT) due to physical distance. This is a deal-breaker for real-time applications like voice AI or interactive coding assistants. Furthermore, EU-regulated teams in healthcare or manufacturing face strict GDPR requirements on where personal data is processed, alongside AI Act documentation obligations for the systems they build.
Lyceum provides a European alternative, with GPU capacity in European data centers in Spain, Paris and the Nordics, billed per second with no base fee. Provisioning capacity there lets you deploy inference endpoints close to your European end-users. Lyceum's scheduling product predicts memory use and runtime within a node and places each job on the right GPU, which reduces the over-provisioning that drives waste on unmanaged instances. Using an OpenAI-compatible API, you can transition from US-based providers to EU-sovereign infrastructure with zero code changes, which keeps inference inside European data centers and cuts network round-trip time for EU users.
Sources
[1] NVIDIA TensorRT-LLM Documentation; [2] vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention; [3] NVIDIA Technical Blog; [4] NVIDIA Dynamo announcement, 18 March 2025; [5] NVIDIA Dynamo 1.0 enters production, 16 March 2026
Frequently Asked Questions
How do I choose between an H100 and an A100 for inference?
The H100 is significantly faster for inference due to its Transformer Engine, which supports FP8 precision. If your budget allows, the H100 provides better price-performance for production workloads. The A100 remains a solid choice for smaller models or when H100 availability is limited.
What is 'Scale to Zero' and how does it affect latency?
Scale to Zero allows your inference endpoint to shut down when not in use, saving costs. The trade-off is 'cold start' latency, where the first request after an idle period waits for the model weights to be loaded back into GPU memory. That delay scales with model size.
Why should I care about GDPR for LLM inference?
If you process personal data of EU citizens, GDPR requires strict data protection measures. Using US-hosted providers can lead to legal uncertainty. EU-hosted providers like Lyceum process inference in European data centers and do not retain prompts or outputs after processing, which simplifies the documentation your legal team needs.
Can I run multiple models on a single GPU?
Yes, if the total VRAM of the models and their KV caches fits within the GPU's capacity. However, sharing a GPU can lead to 'noisy neighbor' issues where one model's traffic spikes increase the latency of the other.
What is continuous batching?
Continuous batching is an optimization where the inference engine inserts new requests into the current processing cycle as soon as a token is generated, rather than waiting for the entire previous batch to complete. This significantly increases throughput and reduces average latency.
Lyceum Technology