Determining the exact number of GPUs for a training run is often the difference between a successful deployment and a wasted six-figure budget. At Lyceum, we see teams struggle with Out-of-Memory (OOM) errors or, conversely, paying for idle H100 clusters that they cannot fully saturate. The decision is not just about raw TFLOPS; it is about the interplay between model weights, optimizer states, and the communication overhead of your interconnect. Whether you are a startup building a niche LLM or an enterprise scaling internal RAG systems, understanding the physics of VRAM and the reality of distributed training is critical for maintaining a competitive edge in the European AI landscape.
GPU Selection Sizing 7 min read
How Many GPUs for Model Training? A Practical Scaling Guide
Throwing more hardware at a model does not always lead to faster convergence. We break down the math behind GPU scaling to help you avoid over-provisioning and maximize training efficiency while maintaining data sovereignty.
Caspar Lehmkühler
January 26, 2026 · Head of Product at Lyceum Technology
Last updated August 3, 2026
The VRAM Trap: Why Model Size is Only the Beginning
The most common mistake in GPU estimation is looking only at the model's parameter count. If you are running a 7B parameter model in FP16 precision, the weights alone take up 14GB of VRAM. However, training is not inference. You must account for gradients, optimizer states, and activations, which routinely quadruple that requirement. For a standard Adam optimizer setup, you typically need 16 to 20 bytes of memory per parameter. This means a 7B model requires roughly 112GB to 140GB of VRAM just to begin training without aggressive optimization.
To solve this, engineers use techniques like Fully Sharded Data Parallelism (FSDP) or DeepSpeed ZeRO. These methods distribute the model states across multiple GPUs, allowing you to train models that would never fit on a single card. If you are targeting a 70B model, even an 80GB H100 cannot hold the weights and optimizer states alone. You are looking at a minimum of 16 GPUs just to fit the model into memory, before even considering the batch size needed for stable convergence.
- Weights: 2 bytes per parameter (FP16/BF16)
- Gradients: 2 bytes per parameter
- Optimizer States: 12 bytes per parameter (for Adam)
- Activations: Variable, depends on sequence length and batch size
At Lyceum, our scheduling product's memory and runtime prediction handles this math within a single node. It analyzes your model architecture and helps you pick a GPU whose memory fits the run, so you avoid the dreaded OOM error while keeping your batch size high enough for efficient throughput. Prediction across multiple nodes, beyond 4 GPUs per node, is still in development.
Scaling Laws and the Time-to-Train Equation
Once you have solved the memory problem, the next question is speed. How long are you willing to wait? Epoch AI's January 2025 analysis of frontier training runs attributes about 40% of compute growth to larger clusters and only about 21% to faster hardware [4], which makes efficient scaling a necessity rather than a luxury. The relationship between the number of GPUs and training time is theoretically linear, but in practice, you hit a wall of diminishing returns known as communication overhead.
Consider a scenario where you need to train a 7B model on 1.4 trillion tokens. Using the standard 6ND compute estimate, that run requires roughly 6 x 10^22 FLOPs. On a single H100, this would take years. By scaling to a cluster of 64 H100s, you can reduce this to a few weeks. However, as you move from 8 GPUs (one node) to 64 or 128 GPUs (multi-node), the speed of your interconnect can become a primary bottleneck for large-model training. Without InfiniBand or RoCE, your GPUs will spend more time waiting for data from their peers than actually performing matrix multiplications.
For most European startups, the sweet spot for fine-tuning is often 8 to 32 GPUs. This range provides a significant speedup without the massive complexity and cost of managing thousand-node clusters. If you are training a foundation model from scratch, you are looking at the scale of 512 to 2,048 GPUs, similar to the infrastructure used for Llama 3.1, which Meta reports was trained on over 16,000 H100s for its largest variant [5].
Distributed Training Strategies: DP vs. FSDP vs. PP
Choosing the right number of GPUs also depends on your chosen parallelism strategy. Not all workloads scale the same way. We categorize these into three main frameworks:
- Data Parallelism (DP): Each GPU has a full copy of the model and processes a different slice of the data. This is simple but limited by the memory of a single GPU.
- Fully Sharded Data Parallelism (FSDP): This shards the model weights, gradients, and optimizer states across all available GPUs. It is the current industry standard for training large models on 2026-era hardware because it maximizes VRAM utilization.
- Pipeline Parallelism (PP): Different layers of the model are placed on different GPUs. This is useful for extremely large models but introduces "bubbles" or idle time where GPUs wait for the previous stage to finish.
If your model is under 10B parameters, simple Data Parallelism across 4 to 8 GPUs is often the most developer-friendly approach. For anything larger, FSDP is mandatory. Lyceum's Serverless Execution-Training jobs handle these configurations for you, so your workload runs sharded in European data centres without you having to write complex distributed boilerplate by hand.
The Infrastructure Bottleneck: NVLink and Interconnects
You cannot just count GPUs; you have to count the wires between them. A cluster of 128 GPUs connected via standard 10Gbps Ethernet will perform significantly worse than 32 GPUs connected via NVLink and 400Gbps InfiniBand. NVIDIA's Blackwell (B200) architecture pushed the boundaries of this with fifth-generation NVLink, which NVIDIA states delivers 1.8TB/s of bidirectional throughput per GPU across up to 576 GPUs [1].
For enterprise IT leaders, the decision often comes down to the "Node" unit. HGX H100 systems are offered with 4 or 8 GPUs. These 8 cards communicate at lightning speeds internally. The moment your training job requires a 9th GPU, you cross the multi-node threshold. This is where latency spikes and throughput can fall off measurably if your cloud provider has not optimized their network fabric. This is why Lyceum focuses on high-performance, low-latency clusters in European data centres in Spain, Paris and the Nordics. The largest single-customer deployment running today is around 200 GPUs, with a 1,000-GPU deployment in build.
Sovereignty and the Cost of Inefficiency
In the European context, the number of GPUs you use is also a matter of data sovereignty and regulatory compliance. Moving massive datasets to US-based hyperscalers often introduces legal friction and latency. By utilizing a European GPU cloud, you keep processing in European data centres while accessing the same H100 and B200 hardware available globally. Efficiency here is not just about TFLOPS; it is about the total cost of ownership (TCO).
We have seen teams over-provision heavily because they feared OOM errors. Profiling your model's memory footprint during a short trial run shows how much of each card the job actually uses. If profiling shows you are only using 40GB of an 80GB H100, you can downscale your instance or increase your batch size to get more value out of every dollar spent. Radical transparency in hardware utilization is the only way to build sustainable AI companies in 2026.
Conclusion: Finding Your Compute Equilibrium
There is no universal number for GPU training. A 7B model might need 4 GPUs for a quick fine-tuning session, while a 400B model requires a small army of accelerators. The goal is to find your compute equilibrium: the point where adding more GPUs still yields a proportional decrease in training time without being throttled by your network or your budget.
Start by calculating your memory requirements using the 16-20 bytes per parameter rule. Then, determine your deadline. If you need results in days, scale horizontally across nodes with high-speed interconnects. If you are on a tighter budget, focus on maximizing the utilization of a single 8-GPU node using FSDP. At Lyceum, we are here to ensure that the infrastructure is the last thing you have to worry about, providing the sovereign, high-performance compute you need to lead the next wave of AI innovation.
Sources
[1] NVIDIA Blackwell Platform Arrives to Power a New Era of Computing; [2] Hugging Face: Model Memory Calculator; [3] arXiv: Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM; [4] Epoch AI: Training compute growth is driven by larger clusters, longer training, and better hardware, 8 January 2025, read 3 August 2026; [5] Meta AI: Introducing Llama 3.1, 23 July 2024, read 3 August 2026
Frequently Asked Questions
What is the difference between training and fine-tuning GPU requirements?
Training from scratch requires processing trillions of tokens and usually demands hundreds of GPUs for weeks or months. Fine-tuning involves a much smaller dataset and can often be completed on 1 to 8 GPUs in a matter of hours or days, depending on the model size.
Why does Lyceum emphasize European sovereignty?
Processing stays in European data centres in Spain, Paris and the Nordics, which supports your own compliance work and gives you low-latency access to compute without trans-atlantic data transfers. Controller obligations under GDPR and any EU AI Act duties remain with you as the customer.
How does batch size affect the number of GPUs I need?
A larger batch size generally improves training stability and throughput but requires more VRAM. If your desired batch size exceeds the memory of your current GPU cluster, you must either add more GPUs or use techniques like gradient accumulation.
What is the 'communication tax' in distributed training?
The communication tax refers to the time GPUs spend synchronizing gradients and weights across the network. As you add more GPUs, this overhead increases. High-speed interconnects like InfiniBand are designed to minimize this tax.
Can I use Lyceum's tools with my existing cloud provider?
Our orchestration layer and VS Code extension are designed to be flexible, though they are built around Lyceum's platform and do not submit jobs to another provider's cloud.
Lyceum Technology