The Evolution of Numerical Precision in Deep Learning

Deep learning precision has evolved through several generations, constantly trading a degree of numerical accuracy for computational efficiency. For years, FP32 (32-bit floating point) served as the undisputed standard for training neural networks. It stores values using 8 exponent bits and 23 mantissa bits, providing immense precision but requiring substantial memory and memory bandwidth. Memory bandwidth, rather than raw compute, often dictates the speed of large language model training. When data must be moved from high-bandwidth memory to the compute cores, smaller data types move faster. FP32 requires 4 bytes per parameter, severely limiting how fast data can be fed to the tensor cores.

The Shift to 16-Bit Formats

As models scaled into the billions of parameters, the industry shifted toward 16-bit formats. FP16 and BF16 reduced memory requirements by half. BF16, in particular, became the default for large language models because it retains the 8 exponent bits of FP32, offering the same dynamic range and generally avoiding the loss scaling that early FP16 training runs needed to stop gradients underflowing to zero. This transition allowed researchers to train larger models on the same hardware footprint, but the exponential growth of model sizes quickly consumed these gains.

The 8-Bit Breakthrough

FP8 takes this reduction one step further by utilizing only 8 bits total. This format cuts compute and memory requirements roughly in half compared to BF16 while maintaining production-quality accuracy. However, squeezing complex neural network representations into 8 bits requires specialized hardware and intelligent software management to prevent the model from collapsing during training. The transition to FP8 is not merely a software trick. It represents a fundamental co-design of hardware and software, spearheaded by the NVIDIA Hopper architecture. By reducing the bit width, the GPU can store twice as many parameters in the same VRAM space, drastically reducing the need for complex tensor parallelism, which is normally kept within a single node, with pipeline parallelism used across servers. This evolution from 32-bit to 8-bit precision highlights a broader trend in artificial intelligence infrastructure, where efficiency gains at the hardware level unlock new possibilities for model scaling and deployment.

The Mechanics of FP8 on Hopper Architecture

The NVIDIA H100 GPU introduced native hardware support for FP8 through its Hopper architecture. Unlike previous formats, FP8 is not a single specification. NVIDIA implements two distinct FP8 variants, each optimized for different phases of the training loop.

Dual Formats: E4M3 and E5M2

The first format is E4M3. This format allocates 4 bits to the exponent and 3 bits to the mantissa. It provides higher precision but a narrower dynamic range. E4M3 is utilized primarily for the forward pass, storing weights and activations where precision is critical for accurate predictions. The second format is E5M2. This format allocates 5 bits to the exponent and 2 bits to the mantissa. It sacrifices some precision to provide a wider dynamic range. E5M2 is deployed during the backward pass, where gradient values can fluctuate wildly and require a broader range to prevent underflow. This dual-format approach allows the architecture to balance the need for precision with the need for dynamic range, adapting to the specific mathematical requirements of each training phase.

The Role of the Transformer Engine

Managing the transition between these formats manually would be an engineering nightmare. To solve this, the H100 features a dedicated Transformer Engine. The hardware dynamically selects the optimal precision for each layer and operation. It scales values up before casting them to FP8 to prevent underflow, executes the matrix multiplication using the high-speed FP8 Tensor Cores, and then scales the results back down. This dynamic scaling ensures that the model maintains the same convergence behavior as a model trained entirely in higher precision. The Transformer Engine operates smoothly under the hood, integrating with major deep learning frameworks to abstract away the complexity of mixed-precision management. By automating the casting and scaling processes, the Hopper architecture allows machine learning engineers to focus on model architecture and hyperparameter tuning rather than low-level numerical stability issues. This hardware-software synergy is what makes FP8 training viable for massive foundation models.

Quantifying the Memory Savings

The primary advantage of FP8 training is the drastic reduction in VRAM consumption. Storing a model's parameters, gradients, and optimizer states requires massive memory allocation. Moving from 16-bit to 8-bit precision cuts the baseline storage requirement for weights and activations by exactly 50%.

Halving the Parameter Footprint

Compared with legacy FP32 mixed-precision setups, FP8 reduces overall memory usage substantially, though the exact figure depends on which tensors stay in higher precision. A 70-billion parameter model provides a clear example of these savings. In BF16, the weights alone consume roughly 140GB of memory. FP8 reduces this to 70GB. That puts the weights of a 70B model inside a single 80GB H100 for inference, with the remaining headroom going to the KV cache. Training is a different budget: gradients, optimizer states and a higher-precision master copy still have to be sharded across multiple GPUs. The savings extend beyond just the static weights.

Activation Memory and Batch Size

During the forward pass, the network must also store activations in memory to calculate gradients later. FP8 halves this activation memory footprint, which is often the largest consumer of VRAM during training with large batch sizes or long sequence lengths. These memory savings unlock several architectural advantages for machine learning engineers. With less memory consumed by weights and activations, you can increase the batch size, directly improving hardware utilization. Furthermore, models that previously required tensor parallelism across eight GPUs can often fit on four. This reduces the collective communication overhead, which is frequently the primary bottleneck in distributed training. By shrinking the memory footprint, FP8 allows teams to train larger models on fewer GPUs, maximizing the return on investment for high-performance compute clusters. The ability to double the batch size or sequence length without triggering out-of-memory errors provides a massive boost to overall training efficiency and model capability.

Throughput Gains and Hardware Utilization

Memory savings directly translate to computational speed. Because FP8 values occupy less space in high-bandwidth memory (HBM3) and the L2 cache, the GPU spends less time moving data and more time executing matrix multiplications. The H100 FP8 Tensor Cores are designed to process these smaller data types at unprecedented speeds.

Accelerating Matrix Multiplications

NVIDIA's NeMo measurements on H100 show 1.30x throughput on Llama 3 8B, 1.43x on Llama 3 70B and 1.53x on Llama 3.1 405B versus BF16, using the current scaling recipe. The performance gap widens on larger models because they involve more matrix multiplications and data movement, both of which benefit substantially from the reduced memory footprint. When data is smaller, the GPU can load it into the tensor cores much faster, keeping the compute units fully saturated. This saturation is critical for achieving high teraFLOPS utilization during large-scale training runs.

Infrastructure Optimization

Maximizing these throughput gains requires infrastructure that does not bottleneck the GPUs. High-performance clusters with self-serve VM provisioning and per-second billing allow infrastructure leads to spin up H100 nodes, test FP8 scaling recipes, and scale down immediately. Lyceum serves H100 capacity from European data centers in Spain, Paris and the Nordics, at $2.79 per GPU-hour for an H100 on-demand VM and $3.59 per GPU-hour for dedicated inference and serverless training, and its scheduling product predicts memory use and runtime within a node so each workload lands on hardware sized to the job. By combining the hardware acceleration of the H100 with highly optimized cloud infrastructure, organizations can drastically reduce the time required to train foundation models. The synergy between FP8 precision and bare-metal performance ensures that every clock cycle is used efficiently, translating technical benchmarks into real-world business value.

Implementation Challenges and Scaling Recipes

Adopting FP8 introduces specific engineering challenges. The reduced dynamic range of 8-bit floats makes models highly susceptible to numerical instability. If gradient values fall below the representable range of E5M2, they underflow to zero, effectively halting the learning process for those parameters.

Managing Numerical Instability

When implementing FP8, engineering teams must avoid common pitfalls. First, you cannot quantize everything. Certain operations, such as LayerNorm and softmax, are highly sensitive to precision loss and must remain in FP32 or BF16. Second, the optimizer typically maintains a high-precision FP32 copy of the master weights, though BF16 recipes that update the 16-bit weights directly keep no separate master copy. Gradient updates during training are often extremely small. If the master weights were stored in FP8, these tiny updates would be lost due to the limited precision. Keeping a master copy in FP32 ensures that small gradient accumulations are accurately recorded.

Choosing an FP8 Scaling Recipe

To mitigate the risks of underflow and overflow, NVIDIA NeMo offers several scaling recipes: delayed scaling, current scaling, sub-channel block scaling and MXFP8. Delayed scaling tracks the maximum absolute values of tensors over several iterations and derives a scaling factor from that history. This factor shifts the tensor values into the safe representable range of the FP8 format before the matrix multiplication occurs. After the operation, the results are scaled back down to their original magnitude. Delayed scaling is computationally efficient because it uses historical data to compute the scaling factor, avoiding the overhead of calculating the maximum value on the fly for every single operation. NVIDIA notes that its robustness suffers on outlier values, and ran the H100 benchmarks above on current scaling, which adapts to each tensor's range immediately. Implementing these recipes correctly is crucial for maintaining model convergence. Without proper scaling, the network will quickly diverge, rendering the training run useless. Engineering teams must carefully validate their scaling configurations before committing to a large-scale FP8 training job.

Optimizing the Software Stack for FP8

Hardware capabilities are only half of the equation. To fully leverage FP8 memory savings, your software stack must be explicitly configured to utilize the Transformer Engine. The open-source ecosystem has rapidly adapted to support these lower-precision formats, but configuration requires precision.

Framework Integration

PyTorch natively integrates with the NVIDIA Transformer Engine, allowing developers to wrap their model layers in an autocast context manager. When enabled, the framework automatically intercepts linear layers, handling the re-casting and scaling and dynamically choosing between FP8 and 16-bit calculations. However, achieving maximum memory reduction requires integrating advanced memory management techniques alongside FP8. The Transformer Engine handles the complex casting between E4M3 and E5M2 smoothly, but the surrounding software architecture must be optimized to prevent bottlenecks. Data loading pipelines, for example, must be fast enough to keep up with the accelerated matrix multiplications.

Combining FP8 with Distributed Training

Combining FP8 with Fully Sharded Data Parallel (FSDP) or DeepSpeed ZeRO allows teams to distribute the FP32 master weights and optimizer states across multiple GPUs. While FP8 halves the memory footprint of the active forward and backward passes, the optimizer states remain a significant memory burden. By sharding these high-precision states, you prevent any single GPU from running out of memory, enabling the training of models exceeding 100 billion parameters on standard H100 clusters. Open-stack transparency is critical here. Standard training frameworks such as PyTorch FSDP, DeepSpeed and NVIDIA NeMo keep training scripts portable, and the same logic applies on the serving side with vLLM, TensorRT-LLM and NVIDIA Dynamo rather than a black-box proprietary engine. This flexibility allows engineering teams to experiment with different distributed training strategies, finding the optimal balance between memory consumption, communication overhead, and raw throughput for their specific model architecture.

Transitioning from BF16 to FP8: A Practical Framework

Moving a production training pipeline from BF16 to FP8 requires a systematic approach to ensure model convergence remains stable. Engineering teams should not flip the switch on a massive training run without validating the numerical stability of the new precision format.

Establishing a BF16 Baseline

Running a small-scale baseline training job in BF16 establishes the ground truth. Record the loss curve, gradient norms, and validation metrics. This baseline serves as the ground truth for your model's expected behavior. It is crucial to capture detailed metrics during this phase, as subtle deviations in the loss curve can indicate underlying numerical issues when transitioning to lower precision. Next, enable FP8 with a Transformer Engine scaling recipe; NVIDIA's own H100 runs used current scaling.

Validating FP8 Convergence

Run the exact same job and overlay the loss curves. In a properly configured setup, the FP8 loss curve should perfectly track the BF16 baseline. If you observe divergence or sudden spikes in the loss, it typically indicates that a sensitive operation, such as a custom activation function or a specific normalization layer, is being incorrectly cast to FP8. After confirming stability on a small scale, gradually increase the batch size. Monitor the VRAM consumption using standard profiling tools like NVIDIA Nsight Systems. You should observe a distinct drop in memory usage, allowing you to push the batch size higher than what was possible in BF16. Finally, evaluate the throughput in tokens per second to quantify the exact speedup on your specific architecture. This methodical validation ensures that you capture the memory savings and throughput gains of FP8 without compromising the integrity of your foundation model. Rushing this process can lead to degraded model quality, negating the computational benefits of the H100 architecture.

The Economics of FP8 Training

The combination of halved memory requirements for the tensors held in FP8 and the 1.43x throughput NVIDIA measured on Llama 3 70B fundamentally alters the unit economics of AI training. By fitting larger models on fewer GPUs and completing training runs faster, teams can drastically reduce their compute spend.

Reducing Compute Spend

A 30-day BF16 training run finishes in about 21 days at 1.43x, but only if the GPU count stays the same. The memory savings are a separate lever: spend them on fewer GPUs and the wall-clock gain shrinks rather than compounds. This acceleration translates directly into lower infrastructure costs and faster time to market for new AI products. Traditional cloud pricing structures often negate these savings. Hyperscaler GPU pricing remains unsustainable for weeks-long training runs, and mandatory block reservations force startups to pay for idle compute. When locked into long-term contracts, the efficiency gains of FP8 do not always result in lower monthly bills.

Sovereign Infrastructure

This challenge is addressed by a different cost base. From European data centers in Spain, Paris and the Nordics, an H100 on-demand VM lists at $2.79 per GPU-hour, and $3.59 per GPU-hour for dedicated inference and serverless training. S3-compatible storage carries no ingress or egress charges, and per-second billing with no subscription or base fee means you pay for exact usage. This model complements the speed of FP8 training, allowing you to spin down resources the moment your accelerated training job completes. For European teams, compliance is just as critical as cost. The platform provides GDPR-compliant processing in European data centers. No training on customer data, ever. Inference prompts and outputs are not retained after processing. A DPA with named sub-processors is available on request, and the data center operators hold ISO certifications at facility level. By combining the technical efficiency of FP8 with the economics of European infrastructure, AI teams can scale training without exhausting their runway. This holistic approach to AI infrastructure ensures that technical breakthroughs at the hardware level deliver tangible business outcomes.

Sources

[1] arXiv: FP8 Formats for Deep Learning (Micikevicius et al., NVIDIA/Arm/Intel); [2] NVIDIA: NVIDIA Hopper Architecture In-Depth (Transformer Engine and FP8); [3] Faster Training Throughput in FP8 Precision with NVIDIA NeMo | NVIDIA Technical Blog; [4] Mixed Precision Training, NVIDIA NeMo Framework User Guide