The Four Pillars of GPU Memory Consumption

To solve memory issues, you must first quantify where the bytes are going. In a standard LLM workload, memory is partitioned into four distinct areas. Model Weights are the most obvious, typically requiring 2 bytes per parameter in FP16/BF16 precision. A 70B model in half-precision requires 140GB of VRAM just to load the weights into memory before a single token is processed.

Optimizer States and Gradients dominate during the training and fine-tuning phases. While weights take 2 bytes, mixed-precision AdamW training requires an additional 12 bytes per parameter [4]: 4 bytes for an FP32 master copy of the weights, plus 4 bytes each for the momentum and variance buffers. This means a 7B model, which seems small, can easily exceed the 80GB limit of an NVIDIA H100 if the training loop is not optimized. Activations are stored during the forward pass to calculate gradients during the backward pass, and their size scales linearly with batch size and sequence length.

The KV Cache is the silent killer of inference performance. As the model generates tokens, it stores the Key and Value vectors for all previous tokens to avoid redundant computations. The cache grows linearly with sequence length and batch size, while the attention computation itself grows quadratically. In long-context windows, the KV cache can consume a large share of total VRAM, often triggering OOM errors mid-generation even if the initial load was successful.

Quantization Strategies: Trading Precision for Throughput

Quantization is the most effective lever for reducing the memory footprint of model weights. By moving from 16-bit (BF16) to 8-bit (INT8) or 4-bit (NF4) precision, you can fit significantly larger models on existing hardware. The bitsandbytes library has popularized 4-bit NormalFloat (NF4) quantization, which brings a 70B model down to roughly 35GB of weights, so it runs on a single A100 80GB with room for a substantial KV cache.

  • FP8 (8-bit Floating Point): Supported natively by NVIDIA H100 and H200 GPUs. It offers a middle ground with minimal accuracy loss and significant speedups in throughput.
  • 4-bit (NF4/AWQ): Reduces memory usage by up to 74.2% compared to FP16, the figure once NF4's quantization constants are counted with double quantization [2]. While there is a measurable impact on perplexity, it is often negligible for task-specific fine-tuning.
  • GGUF/IQ Quantization: Useful for edge deployments or CPU-offloading scenarios, though less common in high-performance data center environments.

When choosing a quantization level, consider the Quantization Tax. While 4-bit quantization is efficient, it can lead to a small drop in benchmark accuracy for reasoning tasks. For finance or healthcare applications where precision is non-negotiable, FP8 or INT8 is often the safer production standard. Lyceum's inference stack builds on vLLM, NVIDIA Dynamo and TensorRT-LLM, which support these precision formats on current NVIDIA hardware.

Advanced Attention and Memory Management

Standard attention is memory-inefficient because it materializes the full attention matrix in VRAM. FlashAttention-3, published in July 2024 for NVIDIA's Hopper generation [1], solves this by using tiling and recomputation to keep the attention calculation within the GPU's SRAM. This reduces memory reads/writes and allows for much longer context windows without the quadratic memory scaling typically seen in older implementations.

For inference-heavy workloads, PagedAttention (the core of the vLLM library) is the industry standard. Traditional memory allocation for the KV cache is contiguous, leading to internal and external fragmentation. PagedAttention treats GPU memory like virtual memory in an operating system, dividing the KV cache into non-contiguous blocks. This cuts KV cache waste from the 60% to 80% typical of contiguous allocation down to under 4%, so many more sequences fit in the same VRAM and throughput rises with them [3].

  1. Identify the bottleneck: Use nvidia-smi or PyTorch Profiler to determine if OOM occurs during model loading or during the forward pass.
  2. Implement Gradient Checkpointing: This technique trades compute for memory by discarding activations during the forward pass and re-calculating them during the backward pass. The paper that introduced it cut a 1,000-layer residual network from 48GB to 7GB for about 30% extra running time, and the same trade applies to transformer activations [5].
  3. Enable Grouped Query Attention (GQA): If you are training a custom model, GQA reduces the number of Key and Value heads, significantly shrinking the KV cache size without sacrificing much model quality.

Distributed Training and ZeRO Redundancy

When a single GPU is insufficient, distributed strategies are required. The DeepSpeed ZeRO (Zero Redundancy Optimizer) ecosystem provides a tiered approach to memory management. ZeRO-1 shards optimizer states, ZeRO-2 shards gradients, and ZeRO-3 shards the model weights themselves across the entire GPU cluster. This allows for the training of models with trillions of parameters that would be impossible to fit on any single node.

For European enterprises operating in regulated sectors, scaling across clusters must be handled with strict data sovereignty. Lyceum runs these jobs in European data centres in Spain, Paris and the Nordics, with the high-bandwidth interconnects (like InfiniBand or RoCE) that ZeRO-3 needs to function without massive latency penalties. Fully Sharded Data Parallel (FSDP) in PyTorch offers a similar benefit, sharding parameters, gradients, and optimizer states to ensure that the memory load is perfectly balanced across your fleet.

Common mistakes in distributed setups include failing to account for CPU RAM offloading. While DeepSpeed allows offloading optimizer states to the CPU, this introduces a massive bottleneck in training speed. It should be viewed as a last resort for fitting a model onto limited GPU resources rather than a primary scaling strategy. Instead, prioritize efficient sharding and pipeline parallelism to keep the workload on the GPU's high-speed memory bus.

Sources

[1] Vijay Thakkar and Fred Oh, Next Generation of FlashAttention, NVIDIA Technical Blog, 11 July 2024; [2] Making LLMs even more accessible with bitsandbytes, 4-bit quantization and QLoRA, Hugging Face, 24 May 2023; [3] vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention, 20 June 2023; [4] ZeRO & DeepSpeed: New system optimizations enable training models with over 100 billion parameters, Microsoft Research Blog, 13 February 2020; [5] Chen, Xu, Zhang and Guestrin, Training Deep Nets with Sublinear Memory Cost, arXiv:1604.06174, 21 April 2016