The Memory Math of Fine-Tuned Inference

Analyzing memory math reveals why serving fine-tuned models is difficult. When you deploy a Large Language Model (LLM), your GPU VRAM is consumed by two primary components: the model weights and the KV (Key-Value) cache.

A 70-billion parameter model illustrates the scale of this problem. In 16-bit precision (FP16), the weights alone consume roughly 140GB of VRAM. If you fine-tune five different variants of this model - perhaps one for legal document parsing, one for code generation, and three for different customer support personas - loading five full copies requires 700GB of VRAM. That exceeds the 640GB an 8x H100 80GB node provides, so you are into a second node before processing a single user request.

The KV Cache Bottleneck

The weights are static, but the KV cache is dynamic. During inference, the model stores the Key and Value states of past tokens to avoid recomputing them for every new token generated. The size of the KV cache scales linearly with the sequence length and the batch size.

  • KV Cache Formula

    2 * sequence_length * layers * hidden_size * batch_size * bytes_per_parameter

For long-context applications, the KV cache can quickly grow larger than the model weights themselves. If you dedicate a GPU to a single fine-tuned model that only receives sporadic traffic, you are wasting massive amounts of expensive memory. According to a Predibase playbook on LLM distillation, serving adapter-based LLMs significantly reduces the cost and complexity of deployment compared to full fine-tuning.

Moving Away from Full Fine-Tuning

The traditional approach of full-parameter fine-tuning creates an unsustainable deployment model. Every new task requires a completely independent set of weights. Managing these separate endpoints leads to severe underutilization of compute resources. When traffic spikes for one variant but remains flat for another, you cannot easily share compute resources across them. The memory footprint becomes the defining constraint of your infrastructure, forcing engineering teams to over-provision hardware just to keep the system stable during peak loads. This is why the industry has rapidly shifted toward parameter-efficient methods that decouple the base model from the task-specific knowledge.

vLLM vs. TensorRT-LLM: Choosing Your Engine

You cannot run production inference using basic Python scripts. You need a dedicated inference engine. Two primary frameworks dominate the landscape: vLLM and TensorRT-LLM. They solve the same problem but take entirely different philosophical approaches to memory management and execution speed.

vLLM: The King of Flexibility

vLLM revolutionized inference by introducing PagedAttention. Traditional KV cache allocation suffers from severe memory fragmentation. The vLLM project measured existing serving systems wasting 60 to 80 percent of memory to fragmentation and over-reservation, against under 4 percent once PagedAttention is in play. PagedAttention solves this by treating the KV cache like virtual memory in an operating system, breaking it into non-contiguous blocks. This allows vLLM to batch significantly more requests concurrently without running out of VRAM.

vLLM is highly dynamic. It supports continuous batching (adding new requests mid-decode) and handles heterogeneous traffic spikes exceptionally well. If you are swapping models frequently or running a wide variety of batch shapes, vLLM is the default choice for maintaining high throughput.

TensorRT-LLM: The King of Raw Speed

TensorRT-LLM, built by NVIDIA, optimizes for raw hardware efficiency. It leans on kernel fusion, CUDA graph capture, and low-precision formats such as FP8, tuned to the exact NVIDIA architecture you run on. The ahead-of-time engine build that once defined it is gone: NVIDIA removed the TensorRT backend in release 1.2.0, leaving PyTorch as the sole execution backend, so there is no separate compile step and no prebuilt engine to ship.

NVIDIA's benchmarking guide walks through tuning TensorRT-LLM for peak throughput with trtllm-bench, and on stable, well-characterized traffic a hardware-specific runtime tuned this way generally edges out a more general one. Treat any cross-engine tokens-per-second figure as hardware-specific and re-measure it on your own GPUs: a 70B model in FP16 needs roughly 140GB for weights alone, so single-A100 numbers imply quantization that changes the comparison. This speed also comes at the cost of flexibility. The configuration is pinned to NVIDIA hardware and to the traffic profile you tuned against, so a drastic change in traffic shape means re-tuning and re-benchmarking before the numbers hold again.

The Lyceum Approach to Inference

At Lyceum, we prioritize open-stack transparency. By standardizing on vLLM and integrating NVIDIA Dynamo, we give engineering teams high-performance inference without the vendor lock-in of proprietary, black-box serving engines. You get the flexibility of open-source with the performance optimizations required for production. This balance is critical when managing multiple fine-tuned variants, as the ability to dynamically allocate memory blocks outweighs the marginal gains of ahead-of-time compilation for most enterprise workloads.

The Multi-LoRA Serving Paradigm

If loading five full copies of a 70B model is financially unviable, what is the alternative? Multi-LoRA Serving provides the solution.

Instead of full-weight fine-tuning, modern teams use Low-Rank Adaptation (LoRA). LoRA freezes the base model weights and trains a small set of adapter weights. The original LoRA paper reports cutting trainable parameters by a factor of 10,000 for GPT-3 175B, and a shipped adapter is typically a fraction of a percent of the base model on disk. In production, you load the massive base model into GPU memory exactly once. When a request arrives, the inference engine dynamically loads the specific LoRA adapter required for that request, injects it into the forward pass, and unloads it.

Handling Rank-Induced Heterogeneity

While multi-LoRA serving is highly efficient, it introduces a new challenge: rank-induced heterogeneity. Not all adapters are the same size. You might have a rank-8 adapter for a simple classification task and a rank-128 adapter for complex reasoning.

Research on serving heterogeneous LoRA adapters (arXiv:2511.22880) finds that state-of-the-art systems co-batch adapters without accounting for rank variability, which produces severe performance skew and forces teams to add GPUs to hold their service-level objectives. Workload-aware placement and routing recovered up to 2x throughput and up to 9x lower Time-to-First-Token (TTFT) on production traces, with up to 50 percent fewer GPUs. The inference engine struggles to batch requests that require different amounts of compute, leading to severe pipeline stalls and inefficient GPU utilization.

Workload-Aware Dynamic Placement

To mitigate this performance degradation, production systems must implement workload-aware dynamic adapter placement. This involves grouping requests by adapter rank and utilizing fast PCIe or NVLink bandwidth to swap adapters from CPU RAM to GPU VRAM in milliseconds. By intelligently scheduling requests that share similar compute profiles, the inference engine can maintain high batch sizes without bottlenecking on the largest adapter in the queue. End-to-end LLM workflows must account for these routing complexities. If your API gateway randomly distributes requests across a cluster without awareness of the underlying adapter ranks, you will inadvertently trigger constant context switching, negating the cost benefits of the multi-LoRA architecture.

Infrastructure Strategy: The Hyperscaler Cost Trap

The software stack is only half the battle. The hardware you run it on dictates your unit economics. Many engineering teams start by renting dedicated GPUs from public cloud hyperscalers, only to find their budgets decimated within months.

The Utilization Problem

Low utilization drives up infrastructure costs. Auto-scaling GPUs on public clouds is notoriously difficult. Providers often require block reservations for high-end hardware like H100s. This means you pay for 24/7 uptime, even when your inference traffic drops to zero overnight. If your cluster utilization hovers around 40 percent, which is common for bursty AI workloads, every hour of useful work costs you two and a half times the list rate, because you rent 2.5 hours to get one. Avoiding these idle costs requires tight integration between the serving layer and the underlying infrastructure. When you are forced to reserve static blocks of compute, the financial benefits of efficient multi-LoRA serving are completely erased by the hardware bill.

Sovereignty and Per-Second Billing

For European teams, data residency adds another layer of friction. Fine-tuned models often process proprietary company data, personally identifiable information (PII), or regulated financial records. Sending this data to non-EU servers pulls you into GDPR Chapter V transfer mechanisms and, in regulated sectors, national localization rules; the AI Act itself imposes no data-residency requirement. Compliance is not something to trade away for cheaper compute.

Lyceum runs GPU infrastructure in European data centres in Spain, Paris and the Nordics. H100 on-demand VMs list at $2.79 per GPU-hour, with dedicated inference and serverless training at $3.59 per GPU-hour, billed per second with no subscription and no base fee. S3-compatible storage is free of ingress and egress charges. Per-second billing means idle hours are not billed as if they were serving traffic, which is where most of the utilization gap turns into cost. By combining per-second billing with efficient multi-LoRA serving, enterprise teams can deploy dozens of specialized models without the massive financial overhead typically associated with generative AI production environments.

Common Production Mistakes

Even with the right engine and infrastructure, deployment can fail if you ignore operational realities. Moving from a local testing environment to a highly available production endpoint requires a fundamental shift in how you manage resources. Teams often encounter several common pitfalls when moving fine-tuned models to production:

Ignoring Cold Starts

Loading a 40GB model from object storage into GPU VRAM takes time. If your scale-to-zero configuration does not account for this, your first user will experience a 30-second timeout. You must implement distributed caching or keep the base model loaded while only swapping adapters. Adapter-based architectures are the practical mitigation: the base model stays resident and a 200MB adapter moves from host memory to VRAM in milliseconds, against minutes to pull a full set of weights from object storage.

Over-provisioning for Peak Traffic

Reserving static blocks of GPUs instead of building a queue-delay autoscaling system leads to massive cost overruns. Your infrastructure should scale based on concurrent requests and queue depth, not arbitrary CPU metrics. Relying on traditional web server scaling metrics will cause your GPU cluster to scale too late or over-provision unnecessarily. You need custom metrics tied directly to the inference engine's internal KV cache utilization.

Overlooking Compliance and Data Residency

Treating infrastructure as a commodity can lead to severe legal exposure. Ask your provider to state plainly which certifications it holds, which claims are self-asserted positions, and which items sit on a roadmap, before deploying models that handle sensitive user data. Many teams fine-tune models specifically to handle proprietary internal documents. If those models are served on infrastructure that routes traffic outside of your legal jurisdiction, you may take on GDPR Chapter V international-transfer obligations and, in some sectors, national localization rules. Neither the GDPR nor the AI Act imposes a general EU data-residency requirement. Lyceum runs GPU compute in European data centres in Spain, Paris and the Nordics, and provides a DPA with named sub-processors on request.

Model Distillation as an Alternative to Fine-Tuning

While fine-tuning is the standard approach for adapting a model to a specific domain, it is not the only method available. For teams struggling with the computational overhead of serving massive 70B or 100B parameter models, model distillation offers a powerful alternative. Distillation involves training a smaller, more efficient student model to replicate the behavior and output quality of a much larger teacher model.

The Economics of Distillation

Knowledge distillation was introduced as a way to compress what a large model or an ensemble knows into a single model that is much easier to deploy, and that is what alters your production economics. Instead of serving a massive model that requires multiple GPUs to hold the weights, you can distill the necessary knowledge into a 7B or 8B parameter model. At 16-bit precision a 7B model is roughly 14GB of weights, so it fits on a single, less expensive GPU, cutting your hourly infrastructure costs.

Distillation is particularly effective for narrow, well-defined tasks. If your application only needs to extract JSON entities from legal contracts, you do not need the broad, general knowledge of a 70B model. By generating a high-quality synthetic dataset using the larger model, you can train a smaller model to achieve parity on that specific task.

Combining Distillation with LoRA

The most advanced production setups combine both techniques. You can distill a large model into a smaller base model, and then use multi-LoRA serving on that smaller base model to handle various sub-tasks. This hybrid approach maximizes both memory efficiency and task performance. The smaller base model ensures that the baseline KV cache and weight memory requirements remain low, while the LoRA adapters provide the flexibility to serve multiple user personas or specific customer configurations without deploying separate endpoints. By leveraging these techniques on Lyceum infrastructure, engineering teams can achieve exceptional inference speeds while maintaining strict control over their cloud budgets.

Optimizing End-to-End LLM Workflows

Serving the model is only one component of a production AI system. To achieve reliable performance, engineering teams must optimize the entire pipeline, from the moment a user submits a prompt to the final token generation. Bottlenecks often occur outside of the inference engine itself.

Data Preprocessing and Tokenization

Before a request ever reaches the GPU, the input text must be tokenized. In high-throughput systems, inefficient Python-based tokenizers can become a severe CPU bottleneck, starving the GPU of work. Production systems must utilize highly optimized, compiled tokenizers and handle preprocessing asynchronously. If your GPU is waiting for the CPU to format a prompt template, you are wasting expensive compute cycles. Implementing a dedicated preprocessing service that feeds a continuous stream of ready-to-compute tokens into the inference engine is critical for maximizing hardware utilization.

API Gateway and Request Routing

When serving multiple fine-tuned models via LoRA adapters, the API gateway plays a crucial role. It must intelligently route requests based on adapter availability and current GPU memory states. If the gateway uses a round-robin approach, it may send a request for a specific adapter to a node that has just unloaded it, forcing a costly reload from CPU memory. A workload-aware router tracks which adapters are currently active in the VRAM of specific nodes and directs traffic accordingly. This minimizes context switching and ensures that the inference engine can maintain high batch sizes. Furthermore, implementing robust retry logic and fallback mechanisms at the gateway level ensures high availability. If a specific node experiences a memory out-of-bounds error due to an unexpectedly large KV cache allocation, the gateway must smoothly redirect the request to a healthy node without exposing the failure to the end user. By treating the entire workflow as a cohesive system rather than isolated components, teams deploying on Lyceum can extract the maximum possible performance from their allocated hardware.

Benchmarking and Performance Tuning

Deploying a fine-tuned model without establishing a rigorous benchmarking protocol is a recipe for unpredictable production failures. You must understand how your specific model behaves under various load conditions before routing live user traffic to it. The NVIDIA developer blog on LLM inference benchmarking with TensorRT-LLM provides a blueprint for how teams should approach performance tuning.

Defining Key Performance Indicators

Throughput and latency are the two primary metrics, but they are often at odds with each other. Throughput is measured in tokens per second across the entire system, while latency is typically measured as Time-to-First-Token (TTFT) and Time-Per-Output-Token (TPOT). If you aggressively increase your batch size to maximize throughput, your TTFT will inevitably degrade, leading to a poor user experience for interactive chat applications. Conversely, optimizing purely for TTFT by keeping batch sizes small will result in terrible GPU utilization and high infrastructure costs.

Simulating Realistic Traffic

Benchmarking must simulate your actual production traffic shape. Sending uniform requests of 500 input tokens and 50 output tokens will not reveal how your system handles edge cases. You must generate synthetic workloads that mimic the long-tail distribution of real user prompts. This includes testing how the inference engine handles sudden spikes in concurrent requests and how it manages memory when processing maximum-context documents. By utilizing tools that profile GPU memory bandwidth and compute utilization during these stress tests, engineering teams can identify exactly where the bottlenecks lie. Whether you choose vLLM for its dynamic batching or TensorRT-LLM for its execution speed (noting that its TensorRT ahead-of-time compilation backend was removed in v1.2.0, leaving PyTorch as the sole execution backend), running these benchmarks on Lyceum infrastructure gives you the empirical data needed to provision the correct amount of hardware for your use case.

Sources

[1] Serving Heterogeneous LoRA Adapters in Distributed LLM Inference Systems; [2] LLM Inference Benchmarking: Performance Tuning with TensorRT-LLM; [3] Best practices for distilling large language models; [4] vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention; [5] LoRA: Low-Rank Adaptation of Large Language Models; [6] Distilling the Knowledge in a Neural Network