How to Right Size GPU Instances for ML Workloads

The VRAM Wall: Calculating Your Memory Floor

Memory is the most common failure point in machine learning infrastructure. If your model does not fit into Video RAM (VRAM), it does not run. Period. Calculating your memory floor is the first step in avoiding the dreaded Out-of-Memory (OOM) error. For a standard transformer model, the weights alone take up significant space. In 16-bit precision (FP16 or BF16), each parameter requires 2 bytes. A 7B parameter model needs roughly 14GB just to load the weights into memory.

Calculating Total VRAM Requirements

Loading the model is only the beginning. During training, you must account for optimizer states and gradients. Mixed-precision Adam holds an FP32 copy of the parameters plus momentum and variance, which is 12 bytes per parameter on top of 2 bytes of FP16 weights and 2 bytes of FP16 gradients, or 16 bytes per parameter of model state in total [5]. Your 7B model therefore needs roughly 112GB before you even consider the batch size or activations, which is already past the 80GB of a single H100 or A100. Hugging Face's caching guide warns that the KV cache can occupy a significant portion of memory and become a bottleneck for long-context generation, which is another common cause of unexpected OOM errors in production [2]. Our wider GPU selection guide for ML training covers how the cards compare once that is settled.

• Model Weights: 2 bytes per parameter for FP16/BF16.
• Optimizer States: 12 bytes per parameter for mixed-precision Adam.
• Gradients: 2 bytes per parameter.
• Activations: Varies based on batch size and sequence length.

If you are running inference, the calculation changes. You no longer need space for optimizer states or gradients, but you must account for the KV cache, which grows linearly with sequence length and batch size. For long-context models, the KV cache can easily exceed the size of the model weights themselves. An NVIDIA A100 with 80GB of VRAM is often a better choice for long-context inference than a faster card with less memory.

Compute Throughput vs. Memory Bandwidth

Once you have cleared the memory floor, the next decision is between compute throughput (TFLOPS) and memory bandwidth (GB/s). These two metrics define how fast your workload will actually run. Training workloads are typically compute-bound. They require massive amounts of raw floating-point operations to update weights. In these scenarios, the H100's Transformer Engine, which NVIDIA rates at up to 3,958 TFLOPS of FP8 with sparsity [3], offers a massive leap over the previous generation A100.

Many inference workloads are memory-bound. This is especially true for Large Language Models (LLMs) during the generation phase. The bottleneck is not how fast the GPU can compute the next token, but how fast it can move the model weights from VRAM to the processors. If your memory bandwidth is low, your high TFLOPS will sit idle. The NVIDIA L40S, for example, offers excellent compute performance but lacks the HBM3 stack found in the H100, which NVIDIA rates at 3.35TB/s against the H200's 4.8TB/s of HBM3e [3][4]. That makes the L40S less ideal for certain high-throughput LLM serving scenarios.

Consider the following scenario: You are deploying a real-time translation service. Low latency is your primary KPI. In this case, choosing an instance with high memory bandwidth is more critical than raw TFLOPS. If you are pre-training a model from scratch, the raw compute power and interconnect speed become the dominant factors. Once you scale beyond a single node, interconnect speed frequently matters more than the individual GPU's speed.

Interconnects: Why NVLink Changes the Math

When a single GPU is not enough, you must scale horizontally. The interconnect is often the silent killer of performance. Standard PCIe Gen4 or Gen5 slots provide a fraction of the bandwidth compared to NVIDIA's proprietary NVLink. If your workload requires frequent communication between GPUs, such as All-Reduce operations in distributed training, PCIe will create a massive bottleneck.

NVLink Bandwidth Advantages for Multi-GPU

An H100 with fourth-generation NVLink provides 900 GB/s of total GPU-to-GPU bandwidth, which NVIDIA puts at about seven times PCIe Gen 5 [1]. Compare this to the 128 GB/s provided by a PCIe Gen5 x16 slot, roughly 64 GB/s in each direction. If you are running a multi-GPU setup without NVLink, your GPUs will spend a significant portion of their time waiting for data to transfer, leading to poor scaling efficiency. In practice, scaling efficiency can drop sharply on PCIe-based clusters for large-scale training tasks.

1. Single-Node Training: Use NVLink-enabled instances to ensure maximum throughput between the 8 GPUs typically found in a HGX baseboard.
2. Multi-Node Training: Look for InfiniBand or high-speed Ethernet (400Gbps+) with RDMA support to minimize latency between servers.
3. Inference Clusters: If you are using model parallelism (tensor or pipeline) to fit a large model across multiple GPUs, NVLink is still highly recommended to maintain low latency.

Lyceum's scheduling product predicts memory use and runtime within a node and places jobs on hardware that fits them. For multi-GPU work, select an NVLink-enabled SXM instance so the interconnect does not become your bottleneck. For European startups, this level of control matters when competing with well-funded US counterparts while keeping data in European data centres.

Sovereignty and the Strategic Choice of Location

Right-sizing is not just about hardware specs; it is about where that hardware lives. For European enterprises, the legal and strategic implications of data residency are becoming as important as TFLOPS. Running sensitive ML workloads on US-based hyperscalers can introduce compliance risks under GDPR and the EU AI Act. When you choose a GPU instance, you must consider the entire lifecycle of your data.

A sovereign European cloud keeps your training data, model weights, and logs within European jurisdiction. This is a strategic advantage, not just a legal checkbox. It allows you to build trust with your customers and ensures that your intellectual property is protected by local laws. Lyceum provides this capacity from European data centres in Spain, Paris and the Nordics, combining high-performance NVIDIA hardware with European operations.

The cost of data egress from major hyperscalers can be a hidden trap. If you train your model in one cloud but need to move it to another for production, the transfer fees can be staggering. By using a specialized AI cloud like Lyceum, you avoid these predatory pricing models. We believe in radical transparency: you should know exactly where your data is processed and exactly what you are paying for, without the obfuscation common in the industry.

Cost Optimization and Automated Prediction

Cost is the final factor in right-sizing. The most powerful GPU is not always the most cost-effective. For many fine-tuning tasks, an A100 or even an L40S might provide better value per dollar than an H100. Match the hardware to the specific precision requirements of your task. If your model supports FP8, the H100's efficiency is unmatched. If you are stuck with FP32 for legacy reasons, you are wasting the H100's potential.

Common cost management mistakes:

• Over-provisioning VRAM: Renting an 80GB card for a model that only needs 24GB.
• Ignoring Spot Instances: Failing to use interruptible capacity for non-time-sensitive training runs.
• Manual Configuration: Spending hours of expensive engineering time manually tuning batch sizes and hardware settings.

We built memory and runtime prediction into our scheduling product to solve this. This tool analyzes your model architecture and workload type to recommend the optimal instance size. It takes the guesswork out of the process, ensuring you have enough VRAM to avoid OOM errors while maximizing use. In an era where GPU availability is often constrained, being able to run your workload on a wider variety of hardware, not just the latest flagship cards, is a major operational advantage.

Decision Framework: Choosing Your Instance

This decision framework simplifies the selection process based on the primary constraint of the workload. Start by identifying if your task is memory-constrained, compute-constrained, or communication-constrained. This will immediately narrow down your hardware choices and prevent expensive misconfigurations.

The hardware landscape changes quickly. The release of the NVIDIA Blackwell architecture in late 2024 and its rollout through 2025 has shifted the baseline for what constitutes high performance. However, for many enterprise applications, the reliability and availability of the Hopper (H100) and Ampere (A100) generations remain the gold standard. The goal is not to have the newest chip, but the one that delivers the best results for your specific budget and compliance requirements.

Sources

[1] NVIDIA, NVIDIA Hopper Architecture In-Depth, 22 March 2022, read 4 August 2026; [2] Hugging Face, Transformers documentation, Cache strategies, read 4 August 2026; [3] NVIDIA, H100 Tensor Core GPU specifications, read 4 August 2026; [4] NVIDIA, H200 Tensor Core GPU specifications, read 4 August 2026; [5] S. Rajbhandari et al., ZeRO: Memory Optimizations Toward Training Trillion Parameter Models, arXiv:1910.02054, read 4 August 2026