The 2026 Open-Weight Landscape: Llama 3, Mistral, and Qwen

The open-weight ecosystem has consolidated around three major model families, each optimized for different production use cases. This migration allows engineering teams to manage their own infrastructure while maintaining data privacy and reducing operational overhead. This shift marks a significant departure from relying solely on proprietary hyperscaler APIs, which often obscure the underlying model mechanics and limit customization. By adopting open-weight architectures, developers can fine-tune models on proprietary datasets, ensuring that the AI aligns perfectly with their specific business logic. Lyceum publishes this article and competes in this market.

Meta's Llama 3 Family

Llama 3.3, particularly the 70B parameter version, has established itself as the industry standard for general reasoning and complex instruction following. It offers a massive 128k context window and performance that rivals proprietary frontier models. However, its size demands significant VRAM, making it a heavy lift for single-GPU deployments without aggressive quantization. Teams deploying Llama 3.3 70B must carefully consider their hardware strategy, often requiring multi-GPU setups or advanced serving engines to handle the memory footprint effectively. Despite the hardware requirements, the model's ability to handle nuanced tasks makes it a favorite for enterprise applications requiring deep contextual understanding.

Mistral AI Efficiency

Mistral continues to dominate the efficiency category within the open-weight landscape. Mistral Small 3, featuring a 32k context window, punches significantly above its weight class. It offers exceptional multilingual support and fast inference speeds on modest hardware. For edge deployments or resource-constrained environments, Mistral remains a top choice. Developers appreciate its ability to deliver high-quality outputs without the massive infrastructure overhead required by larger models. This makes Mistral particularly attractive for startups and teams building lightweight, responsive applications where latency is a primary concern.

Alibaba's Qwen Series

Qwen 2.5, with its 7B, 32B, and 72B variants, is still widely deployed for coding, math, and high-throughput applications, but the current generation is Qwen3, released in April 2025: six dense models from 0.6B to 32B plus two mixture-of-experts variants, Qwen3-30B-A3B and Qwen3-235B-A22B, with context lengths up to 128k, and Alibaba itself recommends serving them with vLLM or SGLang. Qwen3 32B hits the balance many engineering teams want. Its FP8 weights are roughly 32GB, so it fits on a single 48GB card with room for the KV cache, while reasoning quality rivals much larger models. The Qwen architecture is highly optimized for rapid token generation, making it ideal for real-time chat applications and complex data processing pipelines where latency is a critical factor. Its proficiency in non-English languages expands its utility for global deployments.

VRAM Requirements and the Impact of Quantization

Memory management remains the primary bottleneck in large language model serving. A model's parameter count directly dictates its VRAM footprint, but quantization changes the math entirely. Understanding these dynamics is crucial for optimizing infrastructure costs and performance, especially as models continue to grow in complexity.

Full Precision Versus Quantization

At full precision (FP16), a 70B parameter model like Llama 3.3 requires approximately 140GB of VRAM just to load the weights. This does not include the KV cache needed for context, forcing teams into expensive multi-GPU setups. However, utilizing FP8 quantization drastically alters this requirement. By reducing the precision of the weights, a 70B model's footprint shrinks significantly, delivering massive VRAM savings. This allows it to run on a single NVIDIA H100 (80GB), which supports FP8 natively, with room left for the KV cache. An A100 80GB can hold FP8 weights but has no native FP8 tensor cores, so the throughput gain does not carry over. Recent documentation on vLLM multi-GPU setups highlights that running Tensor-Parallel with FP8 on H100 instances provides exceptional cost-efficiency and performance, making massive models economically viable for smaller teams.

Managing the KV Cache

The context window directly impacts memory usage through the KV cache, which makes memory management techniques critical. The vLLM team's PagedAttention post of 20 June 2023 reports that existing serving systems waste 60 to 80 percent of memory to fragmentation and over-reservation, while PagedAttention keeps waste under 4 percent, so effectively all of the reserved cache does useful work. This matters most in high-concurrency environments. When combined with FP8 quantization, inference speed can increase significantly with a minimal drop in output quality.

Baseline VRAM Provisioning

Use this baseline for quantized deployments when provisioning infrastructure to ensure optimal performance without overspending:

7B to 14B Models

(e.g., Qwen 2.5 7B, Llama 3 8B): 16GB to 24GB VRAM. A single 24GB-class data center GPU is typically sufficient; on Lyceum the entry option is the L40S at 48GB.

32B Models

(e.g., Qwen3 32B): 32GB to 48GB VRAM at FP8, because the weights alone are about 32GB. A single L40S 48GB leaves room for the KV cache on top of them.

70B+ Models

(e.g., Llama 3.3 70B): 70GB to 80GB VRAM at FP8. A single H100 80GB is recommended for the weights and a large context window; an A100 80GB holds the same weights but has no native FP8 tensor cores.

Throughput and Latency: vLLM vs. TensorRT-LLM

Raw model size is only one factor in the performance equation. Your choice of inference engine dictates actual production speed and resource utilization. In 2026, the standard stack relies heavily on open-source engines like vLLM and NVIDIA's TensorRT-LLM to maximize hardware efficiency and minimize latency.

Throughput Characteristics on H100 GPUs

On an NVIDIA H100, the smallest models in each family deliver the highest throughput: a 7B or 8B model with optimized batching saturates the GPU long before a 70B model does. This article does not publish a throughput table, because a tokens-per-second figure means nothing without the engine version, batch size, precision, prompt and output lengths, and the date it was measured, and those numbers move with every engine release. vLLM's stable release validates FP8 compatibility across major models like Llama 3.x, Mistral, and Qwen. Setting up vLLM requires careful tuning of environment variables and serving parameters, as detailed in comprehensive guides on LLM serving engine setup. Proper configuration ensures that the GPU compute cores are fully saturated.

The Problem with Proprietary Engines

For larger models, Llama 3.3 70B running on vLLM with FP8 quantization achieves remarkable efficiency. However, achieving these speeds requires deep optimization of the serving stack. Many US-based API providers use proprietary, black-box inference engines to achieve high throughput. This approach creates vendor lock-in and prevents engineering teams from auditing the stack or optimizing it for their specific workloads. You are forced to rely entirely on the provider's internal roadmap, pricing structure, and data handling policies, which may not align with your enterprise requirements.

Lyceum's Open-Stack Architecture

At Lyceum, we take a fundamentally different approach. We utilize an open-stack architecture built on vLLM, NVIDIA Dynamo, and TensorRT-LLM. This transparency ensures customer portability by design. You can inspect the environment variables, adjust the KV cache allocation, and fine-tune the batching parameters to suit your exact needs. NVIDIA Dynamo integration closes the software gap with proprietary engines, giving you enterprise-grade throughput without the lock-in. This open approach guarantees that your inference stack remains flexible, highly performant, and entirely under your control, allowing you to scale operations predictably.

Infrastructure Costs and The Hyperscaler Trap

The hidden cost of AI infrastructure is rarely the GPUs themselves. Instead, it is the restrictive pricing models enforced by major cloud providers. Hyperscalers often require massive block-reservations for high-end GPUs, and their on-demand pricing is notoriously high. For a startup running weeks-long training jobs or sustained 24/7 inference, this burns through runway rapidly and limits the ability to experiment with new model architectures.

The Cost of Hyperscaler Lock-in

Securing an NVIDIA H100 on a major US hyperscaler often involves navigating complex quotas and committing to long-term contracts. Even then, the hourly rates are significantly higher than those offered by specialized providers. Recent community tests have shown that utilizing spot instances for H100 SXM5 GPUs can drop costs dramatically, sometimes reaching rates significantly lower than standard on-demand pricing. However, relying on spot instances from hyperscalers introduces unacceptable volatility for production inference workloads, where uptime is critical and unexpected terminations can degrade the user experience.

The Specialized-Provider Advantage

Specialized infrastructure providers offer a structural cost advantage because their cost base is a focused European footprint rather than a global hyperscaler estate. Lyceum lists H100 at $2.79 per GPU-hour on an on-demand VM and $3.59 per GPU-hour for dedicated inference and serverless training. Billing is per second with no subscription or base fee, the minimum reservation is one month on one server, and S3-compatible storage is free of ingress and egress charges. This transparent pricing model allows teams to forecast their inference budgets accurately without fearing hidden network charges or unexpected billing spikes at the end of the month.

Optimizing Cluster Utilization

Beyond raw hourly rates, cluster utilization is a major cost driver. Industry surveys consistently find that a large share of installed GPU capacity sits idle. To combat this waste, Lyceum built a scheduling product that handles VRAM prediction, runtime estimation, and automatic GPU selection, so each job lands on hardware sized for it. Memory and runtime prediction within a node is a shipped capability; multi-node prediction beyond four GPUs per node is in development. You pay only for what you use, and with scale-to-zero capabilities, your inference endpoints automatically spin down when traffic stops, ensuring maximum capital efficiency and extending your operational runway.

EU Data Sovereignty and Compliance as a Competitive Advantage

For European AI teams, throughput and token generation speed matter less than whether the underlying infrastructure survives a data protection review. Regulated industries such as healthcare, finance, and manufacturing operate under strict legal frameworks. Many of those teams treat US hosting as a procurement blocker for sensitive data such as patient medical records, financial transaction histories, or proprietary factory floor imagery, and any transfer out of the EU needs a Chapter V mechanism plus a transfer impact assessment.

The Risk of US-Based Providers

Most well-known serverless GPU providers are based in the United States and are therefore subject to the CLOUD Act. This legislation allows US authorities to compel access to data stored by these companies, regardless of where the servers are physically located. For European enterprises, that exposure is a risk to document in a transfer impact assessment, and many procurement teams rule these platforms out on that basis rather than on a residency law. Where a sector rule or a customer contract does require local processing, teams need to be able to show where processing happens rather than assert it.

Lyceum's EU-Native Infrastructure

Lyceum provides a different solution through an EU-native inference platform: GDPR-compliant processing in European data centers in Spain, Paris and the Nordics, no training on customer data, and inference prompts and outputs that are not retained after processing. A DPA with the named sub-processor list is available on request, and the data center operators hold ISO certifications at facility level. Be equally direct about the gap: Lyceum holds no ISO 27001, SOC 2 or BSI C5 attestation today and publishes no EU AI Act conformity position. We understand that data sovereignty is not just a legal checkbox, but a core competitive advantage for European businesses building trust with their users in an increasingly privacy-conscious market.

Secure and Dedicated Endpoints

To further guarantee security, we provide dedicated inference endpoints where the underlying machine is exclusively yours. There is no shared tenancy and no risk of cross-customer data leakage. You receive a drop-in, OpenAI-compatible API, meaning you can switch your backend in minutes with zero code changes to your application logic. This allows you to leverage the power of Llama 3, Mistral, or Qwen while running entirely on secure, EU-sovereign infrastructure, giving you the best of both worlds: frontier model performance and absolute data security.

Deploying Your Inference Stack

Getting a powerful open-weight model into production should not require hiring a dedicated DevOps team. While managing your own physical hardware is incredibly painful, involving complex cooling challenges, high maintenance costs, and constant capacity bottlenecks, cloud deployment should be entirely frictionless. Your engineering team should focus on building product features and improving user experiences, not wrestling with CUDA drivers, dependency conflicts, or hardware provisioning.

Frictionless Compute Access

There are multiple ways to access compute depending on your technical requirements. If your team needs raw, root-level access to optimize the operating system, our virtual machines are self-service and start quickly. Lyceum serves GPU compute from European data centers in Spain, Paris and the Nordics, billed per second with no base fee. This means you can secure the H100 or A100 instances you need without waiting in hyperscaler queues, allowing your team to move faster and deploy models on your own schedule.

Streamlined Model Serving

For streamlined model serving, our Dedicated Inference Engine abstracts away the infrastructure complexity. It allows you to host any Hugging Face model or deploy a custom Docker image with ease. As detailed in guides for vLLM multi-GPU setups, deploying via Docker requires specific commands and environment variables to enable features like Tensor-Parallelism and FP8 quantization. Our platform handles these complex configurations automatically behind the scenes. You select the desired GPU, specify the model repository, and receive a secure API endpoint ready to accept traffic immediately.

Flexible Scaling for Any Workload

Lyceum Inference Studio, the serverless option with pre-hosted open models and per-token billing, is live today with 35 models plus 4 smart-routing entries. Whether you are running massive batch OCR processing jobs overnight or handling latency-sensitive medical image segmentation during peak clinic hours, you have the flexibility to scale dynamically. You can scale up instantly during traffic spikes to maintain low latency and scale to zero during idle periods, ensuring you only pay for the compute you actively consume.

Optimizing vLLM Parameters for Maximum Throughput

Achieving high throughput with models like Llama 3, Mistral, and Qwen requires more than a fast GPU. The serving engine must be meticulously configured. vLLM has emerged as the premier choice for this task, but its default settings are rarely optimal for high-traffic production environments. Understanding how to tune its parameters is essential for maximizing throughput and ensuring that your hardware investment is fully utilized.

Key vLLM Environment Variables

A comprehensive guide to vLLM setup highlights several critical environment variables and parameters that dictate performance. One of the most important settings is the maximum number of batched tokens. By increasing the batch size, the engine can process multiple requests simultaneously, significantly boosting the overall tokens per second. However, this must be balanced against the available VRAM, as larger batches consume more memory for the KV cache. Finding the optimal batch size requires iterative testing based on your specific prompt lengths and expected output sizes.

Tuning PagedAttention and Memory Allocation

vLLM utilizes PagedAttention to manage memory efficiently, but administrators must still define the GPU memory utilization ratio. By default, vLLM might reserve a conservative amount of VRAM. For dedicated inference nodes running a single model, increasing this allocation ratio allows the engine to store a larger KV cache. This directly translates to supporting more concurrent users and longer context windows without triggering out-of-memory errors. Proper configuration of these memory parameters ensures that the GPU is fully saturated with useful work, preventing memory fragmentation and maximizing the number of requests handled per second.

Tensor Parallelism for Large Models

When deploying massive models like Llama 3.3 70B, a single GPU is often insufficient. vLLM supports Tensor Parallelism, which splits the model weights across multiple GPUs. Configuring the tensor parallel size correctly is crucial for minimizing inter-GPU communication overhead. By aligning the parallel size with the physical topology of the server, such as an 8-way H100 SXM5 system, teams can achieve near-linear scaling in inference speed. This ensures that large models remain highly responsive under heavy load, providing a smooth experience for end users.

Selecting the Right GPU for Your Workload

The hardware landscape for AI inference is diverse, and selecting the appropriate GPU is a critical decision that impacts both performance and budget. The choice depends heavily on the specific model architecture, the required context window, and the expected concurrency of user requests. Making the wrong choice can lead to severe bottlenecks or wasted resources.

Matching VRAM to Model Size

The primary constraint when choosing a GPU is VRAM capacity. The GPU must have enough memory to hold the model weights and the KV cache. For smaller models like Mistral 7B or Qwen 2.5 7B, entry-level enterprise GPUs like the NVIDIA A10G or consumer-grade RTX 4090 provide excellent performance at a lower price point. These GPUs offer 24GB of VRAM, which is more than sufficient for 7B models, even at full precision, while leaving ample room for a moderate context window and concurrent request batching.

Handling Large Context Windows

When applications require massive context windows, such as analyzing entire codebases or long legal documents, the memory requirements for the KV cache skyrocket. Even if a model's weights fit on a smaller GPU, a large context window will quickly cause out-of-memory errors. In these scenarios, upgrading to GPUs with larger memory pools, such as the A100 80GB or the H100, becomes necessary. The PagedAttention work is explicit that context length is a primary driver of memory consumption during active inference, making it a critical factor when sizing your hardware.

Cost-Performance Trade-offs

Engineering teams must constantly balance cost against performance. While the NVIDIA H100 offers unparalleled throughput and supports advanced features like FP8 quantization natively, it comes at a premium price. For many background tasks or batch processing workloads where latency is not the primary concern, older generation GPUs like the A100, or an L40S, might offer a better cost per token. Lyceum provides a wide range of GPU options, allowing teams to match their hardware precisely to their workload requirements and budget constraints. This flexibility ensures that you are never forced to over-provision expensive hardware for simple tasks.

Sources

[1] vLLM Documentation: Engine Arguments; [2] vLLM Documentation: Parallelism and Scaling; [3] arXiv: The Llama 3 Herd of Models; [4] vLLM Blog: Easy, Fast, and Cheap LLM Serving with PagedAttention, 20 June 2023; [5] Qwen Blog: Qwen3, Think Deeper, Act Faster, 29 April 2025