When you monitor your training jobs and see GPU utilization sitting far below what the hardware can deliver, you are paying for capacity you never use. In high-performance environments, especially those utilizing NVIDIA H100 or A100 GPUs, the hardware is often faster than the software feeding it. This mismatch creates a 'starvation' effect where the GPU completes its work and waits for the next batch. We see this frequently in enterprise AI labs where data pipelines were built for smaller models but never updated for modern compute scales. Fixing this requires a systematic approach to profiling, data orchestration, and memory management to ensure your compute investment is fully leveraged.
Operations Utilisation 8 min read
GPU Utilization Too Low: How to Fix Compute Bottlenecks
Low GPU utilization is rarely a hardware failure. It is almost always a symptom of upstream data starvation or inefficient kernel execution that leaves expensive H100 clusters idling while costs mount. For AI teams scaling on sovereign infrastructure, every wasted cycle represents a delay in model deployment and a direct hit to the bottom line.
Maximilian Niroomand
January 2, 2026 · CTO & Co-Founder at Lyceum Technology
Last updated August 3, 2026
The Data Loading Bottleneck: Feeding the Beast
The most common cause of low GPU utilization is data starvation. If your GPU is waiting for the CPU to finish preprocessing or for the disk to read the next batch, your utilization will plummet. This is particularly prevalent when working with large datasets in regulated industries where data might be stored on slower, encrypted storage systems. In enterprise clusters, I/O wait is one of the largest contributors to idle GPU time, and it is rarely visible without profiling.
Diagnosing Data Pipeline Bottlenecks
To diagnose this, monitor your CPU utilization alongside your GPU. If your CPU is pegged while the GPU is low, your data pipeline is the culprit. You can resolve this by optimizing your DataLoader configuration. In PyTorch, for instance, increasing the num_workers parameter allows for multi-process data loading, which parallelizes the fetching and transformation of data. However, setting this too high can lead to shared memory issues or CPU thrashing.
- Enable Pin Memory: Use
pin_memory=Truein your DataLoader. This enables faster data transfer from the CPU to the GPU by using page-locked memory, bypassing an extra copy step in the host memory. - Prefetching: Use prefetch factors to ensure the next batch is ready before the current one finishes. This masks the latency of data preparation.
- Storage Throughput: Ensure your underlying storage can handle the IOPS required. Moving from standard SSDs to NVMe storage or using high-performance parallel file systems is often necessary for H100 clusters.
We often see teams overlook the impact of complex data augmentations. If you are performing heavy image rotations or text tokenization on the fly, consider moving these operations to the GPU using libraries like NVIDIA DALI. This offloads the burden from the CPU and keeps the GPU pipeline saturated.
Batch Size and Tensor Core Saturation
GPU architecture is designed for massive parallelism. If your batch size is too small, you are not providing enough work to fill the thousands of CUDA cores available. This results in 'under-utilization' even if the GPU is technically active. For modern architectures like Hopper (H100), the Tensor Cores require specific alignment to reach peak TFLOPS.
Gradient Accumulation for Small GPUs
A common mistake is keeping batch sizes small to avoid 'Out of Memory' (OOM) errors. While this prevents crashes, it often leaves a large share of the GPU's compute potential on the table. To fix this, you should find the maximum batch size your memory can support. If your model is too large for a single GPU, consider using Gradient Accumulation. This allows you to simulate a larger batch size by accumulating gradients over multiple smaller steps before performing an optimizer update.
- Power of Two: Always use batch sizes that are multiples of 8, 16, or 32. This aligns with the warp size and memory controller architecture, ensuring efficient memory access patterns.
- Mixed Precision Training: Switching from FP32 to 16-bit precision (FP16 or BF16) effectively doubles your memory capacity and significantly increases throughput. On H100s, using Transformer Engine (FP8) can provide even greater speedups.
- Memory Profiling: Use tools like
torch.cuda.memory_summary()to identify where your VRAM is going. Often, large activations or unoptimized model architectures are the real culprits behind small batch constraints.
The goal is to reach a state where the GPU is the bottleneck, not the data pipeline. If you increase the batch size and the utilization stays low, the issue likely lies in kernel launch overhead or synchronization delays.
Kernel Launch Overhead and CUDA Graphs
In deep learning, a single forward pass consists of hundreds or thousands of individual operations (kernels). Each time the CPU tells the GPU to run a kernel, there is a small amount of overhead. If your kernels are very small or execute very quickly, the time spent launching them can exceed the time spent actually running them. This is known as being 'CPU-bound' on the launch side.
This is a frequent issue with models that have many small layers or complex branching logic. To mitigate this, NVIDIA introduced CUDA Graphs. Instead of launching kernels one by one, a CUDA Graph allows you to 'record' a sequence of operations and launch them as a single unit. This drastically reduces the CPU-to-GPU communication overhead.
Another strategy is Operator Fusion. Compilers like TorchDynamo (part of PyTorch 2.0+) can automatically fuse multiple operations into a single kernel. For example, combining a linear layer, a bias addition, and a ReLU activation into one operation reduces memory trips and launch calls. If you are not using torch.compile() in 2025, you are likely leaving significant performance on the table. In practice, torch.compile often delivers a measurable utilization improvement on standard Transformer workloads without any code changes.
Distributed Training and Communication Latency
When scaling across multiple GPUs or nodes, the bottleneck often shifts from local compute to network communication. In a distributed environment, GPUs must synchronize their gradients (All-Reduce) at the end of every step. If your network is slow or your synchronization strategy is inefficient, your GPUs will sit idle waiting for data from their peers.
This is where the distinction between standard cloud providers and sovereign high-performance clouds becomes clear. Standard cloud instances often lack the high-bandwidth interconnects required for efficient scaling. For H100 clusters, InfiniBand or RoCE (RDMA over Converged Ethernet) is mandatory. Without these, the 'communication-to-computation' ratio becomes unfavorable, and adding more GPUs actually decreases your overall efficiency.
- Check NCCL Performance: Use the
NCCL_DEBUG=INFOenvironment variable to monitor communication logs. Look for large gaps in the timeline where the GPU is waiting for the network. - Gradient Bucketing: Grouping gradients into larger 'buckets' before sending them over the network can reduce the number of individual transfers and improve throughput.
- Pipeline Parallelism: For massive models, use pipeline parallelism to overlap computation and communication. While one layer is computing, the previous layer can be sending its results to the next GPU.
In 2025, we see more enterprises moving toward Sovereign GPU Clouds like Lyceum for dedicated capacity in European data centres, which helps multi-node training jobs maintain high utilization across the entire cluster.
Profiling: Stop Guessing, Start Measuring
You cannot fix what you cannot see. If your utilization is low, the first step should always be profiling. NVIDIA Nsight Systems and PyTorch Profiler are the gold standards for this. These tools provide a visual timeline of your execution, showing exactly where the gaps are.
When looking at a profile, look for 'white space' on the GPU timeline. If the GPU timeline is empty while the CPU timeline is busy, you have a data loading or kernel launch issue. If the GPU is busy but the utilization percentage is low, you likely have a batch size or precision issue. A common mistake is relying solely on nvidia-smi. This tool provides a sampled average of utilization, which can be misleading. A GPU might report full utilization because it is 'active,' but it could be performing inefficient memory copies rather than actual floating-point math.
For a more granular view, use NVIDIA Nsight Compute. This tool allows you to dive into individual kernels to see if they are memory-bound or compute-bound. If a kernel is memory-bound, it means you are limited by the speed of VRAM; if it is compute-bound, you are limited by the TFLOPS of the cores. This level of detail is essential for researchers optimizing custom CUDA kernels or novel model architectures.
The Lyceum Approach: Prediction and Support
At Lyceum, we believe that AI researchers should focus on models, not infrastructure plumbing. Lyceum's scheduling product predicts memory and runtime within a node, and a direct line to the engineering team, within your support tier, helps you tune data prefetching, memory pinning, and communication settings for the specific hardware you are using.
Furthermore, Lyceum runs on European data centres in Spain, Paris and the Nordics, keeping processing in Europe while providing dedicated H100 capacity billed per second with no base fee. The support team works with you on configuration changes, such as adjusting batch sizes or enabling mixed precision, so you get the most out of every compute hour. This proactive approach eliminates the trial-and-error process that typically plagues AI scaling efforts.
Sources
[1] NVIDIA Nsight Systems Documentation; [2] NVIDIA: Data Center GPU Manager (DCGM) Documentation; [3] NVIDIA: DALI Documentation for Accelerated Data Loading Pipelines
Frequently Asked Questions
What counts as 'good' GPU utilization for AI training?
There is no single number worth chasing here. Sustained idle gaps in the profiler timeline are the signal that your data pipeline or model architecture has room for optimization. Keep in mind that 'utilization' as reported by nvidia-smi can be misleading; throughput (samples per second) is the more important metric.
How does mixed precision (FP16/BF16) affect utilization?
Mixed precision reduces the memory footprint of your tensors, allowing for larger batch sizes. It also uses specialized hardware (Tensor Cores) that can perform math much faster than standard FP32 cores. This usually increases utilization by allowing the GPU to process more data in less time.
What is the impact of PCIe bandwidth on GPU utilization?
PCIe bandwidth limits how fast data can move from your CPU/RAM to the GPU. If you are training on a system with PCIe Gen3 instead of Gen4 or Gen5, or if you are using a limited number of lanes, the GPU may spend a significant amount of time waiting for data transfers, leading to low utilization.
Why does my GPU utilization drop to zero periodically?
Periodic drops to zero usually indicate a 'validation' phase where the model is running on a test set, or a bottleneck during checkpoint saving where the training pauses to write the model state to disk. Using asynchronous checkpointing can help mitigate this.
Can I fix low utilization without changing my code?
Sometimes. You can try increasing the batch size via command-line arguments if the script supports it, or moving your data to a faster storage drive (e.g., from HDD to NVMe). However, most significant gains require small code changes like enabling pin_memory or using torch.compile.
Does Lyceum help with GPU utilization automatically?
Partly. Lyceum's scheduling product includes memory and runtime prediction within a node, and a direct line to the engineering team, within your support tier, helps tune parameters like I/O prefetching and memory allocation. A fully automatic optimization engine is not a shipped feature.
Lyceum Technology