The Economics of Serverless (Per-Token) Inference

Serverless inference APIs charge strictly for the compute you consume, measured in input and output tokens. You send a prompt, the provider routes it to a massive, shared GPU cluster, and you pay a fraction of a cent for the response. This model has dominated the early wave of generative AI adoption because it abstracts away the immense complexity of hardware provisioning.

The Appeal of Scale-to-Zero Architectures

Early-stage products, prototypes, and workloads with high variance in traffic benefit immensely from this model's efficiency. You avoid the capital expenditure of reserving hardware and the operational burden of managing container runtimes. When traffic drops to zero overnight, your costs drop to zero. For a team testing a new feature with unpredictable user adoption, the ability to pay only for exact usage is a massive financial safety net.

Where the Serverless Model Breaks Down

However, the serverless model breaks down under three specific conditions that scaling engineering teams inevitably encounter:

  • Sustained Volume and Unit Economics

    At high throughput, the premium baked into per-token pricing outweighs the cost of idle time on a dedicated machine. Serverless providers must charge a markup to cover their own idle capacity and infrastructure overhead. When your application reaches a steady baseline of traffic, you end up paying that premium continuously.

  • Data Sovereignty and Shared Tenancy

    Serverless platforms share infrastructure across thousands of customers. Your inference requests run on GPUs that processed another company's data milliseconds earlier. For EU-regulated teams, shared tenancy makes data residency and isolation hard to evidence, which is exactly what procurement reviews and DPIAs ask you to show.

  • Latency Constraints and Cold Starts

    Scale-to-zero architectures introduce cold starts. Spinning up a containerized model from zero can take tens of seconds depending on the provider and model size. For real-time applications, customer-facing chatbots, or voice agents, this latency is unacceptable and severely degrades the user experience.

The Economics of Dedicated (Per-Hour) Infrastructure

Dedicated infrastructure flips the pricing model entirely. You rent the GPU by the hour or second, and you can push as many tokens through it as the memory bandwidth allows. The cost per token becomes a function of your optimization skills rather than a fixed vendor rate. This shift transforms infrastructure from a variable operational expense into a predictable, manageable line item.

Optimizing Throughput for Lower Costs

To calculate your effective cost per token on dedicated hardware, you need three variables: the hourly GPU rate, the model's tokens-per-second throughput, and your average utilization rate. The beauty of dedicated hardware is that software improvements directly translate to financial savings.

Consider an H100. AWS lists the p5.48xlarge on demand for Linux in US East (N. Virginia) at $55.04 per hour in USD, read on 3 August 2026, which is $6.88 per GPU-hour across the eight H100s in that instance. At that rate a 70B parameter model serving 50 output tokens per second produces 180,000 tokens in a GPU-hour, so compute costs about $0.038 per thousand tokens. Push the same card to 120 tokens per second with continuous batching or quantization and the same hour yields 432,000 tokens, or about $0.016 per thousand. That is a 58 percent reduction with no change to the hardware, and you capture it rather than the provider.

The Risk of Underutilization

The primary risk of dedicated infrastructure is underutilization. Cost per token is the hourly rate divided by the tokens you actually push through the machine, so a card that stands idle for most of the day inflates that figure by the same factor it is idle by, which is how a dedicated instance ends up more expensive than a premium serverless API. If your application only receives traffic for two hours a day, paying for a dedicated instance running 24 hours a day will destroy your unit economics. Engineering teams must implement robust auto-scaling and load-balancing strategies to ensure that dedicated instances remain highly utilized, thereby justifying the shift away from per-token billing.

Furthermore, managing dedicated infrastructure requires a deeper understanding of container orchestration. Teams must deploy frameworks like vLLM or TensorRT-LLM to maximize the hardware's potential. While this requires more upfront engineering effort, the long-term payoff in reduced inference costs per token is substantial for high-growth applications.

Calculating the Break-Even Point

Migrating from serverless APIs to dedicated GPUs depends on a specific mathematical threshold. Engineering leaders cannot rely on intuition when making this transition. There is no single industry threshold worth quoting here, and the arithmetic is short enough to run yourself: dedicated capacity wins once the hourly rate of the instance, divided by the tokens you sustain on it in an hour, falls below the per-token rate on your current API invoice.

Working Out Your Own Threshold

In practical terms, work it out at both ends of your traffic curve. Take the published hourly rate for the instance you would rent, multiply your measured sustained output tokens per second by 3,600 to get tokens per GPU-hour, and divide the first figure by the second. Compare the result with the per-million-token rate you are billed today. Do it at your quietest hour as well as your busiest, because the trough is what decides whether a dedicated machine pays for itself. If your traffic is highly volatile, spiking for a few minutes and then dropping to zero for hours, serverless remains the logical choice.

Above whatever threshold that arithmetic gives you, every additional token processed on a serverless API represents lost margin. On a dedicated instance, those additional tokens carry no extra charge, constrained only by the physical limits of the GPU memory bandwidth.

How Specialized Providers Alter the Equation

This calculation assumes standard hyperscaler pricing, which often includes massive markups for brand recognition and ecosystem lock-in. However, the break-even threshold drops significantly if you source compute from specialized infrastructure providers. When you secure an H100 VM from a specialized provider at a lower hourly rate, dedicated infrastructure becomes cost-effective at much lower traffic volumes.

For example, if a specialized provider offers an H100 instance at competitive rates compared to legacy clouds, the utilization threshold required to beat serverless pricing drops dramatically. Teams can migrate to dedicated hardware earlier in their growth cycle, locking in better margins and superior performance long before they reach massive enterprise scale.

The Hidden Costs of Inference

Comparing sticker prices of an API to the hourly rate of a GPU often ignores secondary costs that inflate monthly bills. When modeling your inference budget, you must account for three hidden factors that frequently catch scaling teams off guard.

Unpacking the Monthly Cloud Bill

  1. Egress Fees

    Major cloud providers charge per gigabyte for outbound data, at rates that vary by region and volume tier. If your application generates 1 TB of output per day, which is common for batch OCR processing, large-scale summarization, or high-volume embedding generation, egress fees alone will add thousands of dollars to your monthly bill. This data transfer tax is rarely factored into initial cost-per-token calculations.

  2. Engineering Overhead

    Managing raw virtual machines requires dedicated platform engineering time. Configuring CUDA drivers, optimizing inference engines like vLLM or TensorRT-LLM, and building robust auto-scaling logic requires specialized, expensive talent. The salary cost of an MLOps engineer spending weeks configuring infrastructure must be amortized into your total cost of ownership.

  3. Compliance Audits and Legal Friction

    For European teams, proving data residency on US-based serverless platforms is often impossible. The engineering hours spent building anonymization proxies, redacting personally identifiable information before it hits an external API, or negotiating custom Data Processing Agreements represent a massive hidden tax on your infrastructure. Legal reviews and compliance audits consume resources that should be spent on core product development.

The Total Cost of Ownership

When evaluating the true cost per token, you must build a comprehensive model that includes these hidden variables. A serverless API might look cheaper on paper, but if it requires extensive data redaction pipelines and incurs massive egress fees, the actual cost to the business is much higher. Dedicated infrastructure from specialized providers often eliminates these hidden fees, offering a more transparent total cost of ownership.

The European Compliance Imperative

AI startups and scale-ups operating in Europe must consider factors beyond unit economics when choosing infrastructure. The vast majority of serverless inference APIs are US-based and US-hosted. They route prompts through proprietary, black-box engines on shared hardware, creating significant legal and regulatory risks.

Navigating the Regulatory Landscape

If you process medical imagery, financial records, or proprietary manufacturing data, routing that information through non-EU servers is a non-starter. European regulation, specifically the General Data Protection Regulation, restricts transfers of personal data outside the EEA under Chapter V. The EU AI Act adds obligations for AI systems but sets no data-residency requirement. You need the ease of an API, but the security of an isolated, sovereign environment.

Relying on US-based hyperscalers often means your data is subject to foreign jurisdictions, even if the data center is physically located in Europe. This legal gray area is unacceptable for enterprise clients who demand absolute certainty regarding where their data lives and who has access to it.

The Sovereign Infrastructure Solution

This is the exact gap specialized European providers fill. We provide an EU inference platform running in European data centers in Spain, Paris and the Nordics. When you deploy a model on a dedicated platform, the machine is exclusively yours. There is no shared tenancy, and processing stays GDPR-compliant in European data centers while you keep the developer experience of a standard API.

Keeping processing inside the EEA removes much of the legal friction attached to international data transfers, although EU law defines no 'sovereign' infrastructure status as such. What you can tell a client is specific rather than blanket: processing in European data centers, with the region named for the model you are running, no training on customer data and no data retained after processing. Check the region model by model, because a serverless catalogue will usually carry some entries served globally rather than from the EU. That specificity is what closes enterprise deals, and it closes them faster than a general assurance does.

Furthermore, sovereign providers offer a level of transparency that black-box APIs cannot match. You have full visibility into the hardware stack, the network routing, and the security protocols protecting your workloads. This transparency is crucial during rigorous enterprise security audits.

How Lyceum Technology Changes the Equation

Lyceum sells GPU compute from European data centers in Spain, Paris and the Nordics, billed per second with no base fee. The list price is lower than a hyperscaler's because the catalogue is narrower: we sell compute, not an entire cloud. That is how we list H100 VMs at $2.79 per GPU-hour on demand, with dedicated inference and serverless training at $3.59 per GPU-hour. Those are published rates for three distinct product modes, so the hardware economics reach your engineering team directly.

A Platform Built for AI Engineering

Our platform is designed specifically for the needs of AI engineering teams who are scaling beyond the prototype phase:

  • Dedicated Inference Endpoints

    Host an open-source large language model from the catalogue in European data centers and serve it via an OpenAI-compatible API. You get a drop-in replacement for your current API with zero code changes required.

  • Zero Egress Fees

    We provide S3-compatible storage free of ingress and egress charges, eliminating the most unpredictable line item in cloud billing. You can generate massive datasets without worrying about bandwidth penalties.

  • Intelligent Scheduling

    Our scheduling product predicts VRAM requirements and estimates runtime within a node, which improves hardware allocation and cuts failed runs.

  • Per-Second Billing

    Per-second billing with no subscription and no base fee; reserved capacity starts at a minimum of one month on one server. That bridges the gap between serverless flexibility and dedicated performance.

Bridging the Gap Between Flexibility and Control

We offer dedicated inference endpoints alongside Lyceum Inference Studio, serverless inference on a catalogue of 35 open-source models billed per token. Whether you need raw SSH access to a virtual machine provisioned quickly or a fully managed inference API, the platform provides the performance you need without compromising on European data sovereignty. Our goal is to make high-performance AI compute accessible and predictable, with GDPR-compliant processing that stays in European data centers.

On-Premise vs. Cloud Inference Economics

When evaluating inference costs, teams eventually face the decision between renting cloud GPUs and purchasing on-premise hardware. While cloud infrastructure offers flexibility, on-premise deployments represent the ultimate form of dedicated compute. Understanding the break-even analysis between these two models is critical for long-term financial planning.

The Capital Expenditure Challenge

Purchasing your own H100 cluster requires a massive upfront capital expenditure. Beyond the cost of the silicon itself, organizations must account for specialized cooling, high-capacity power delivery, and physical security. Furthermore, hardware depreciation cycles in the AI sector are brutally fast. An expensive cluster purchased today may be outclassed by next-generation architectures within two years, leaving you with stranded assets.

On-premise deployments also require a dedicated IT operations team to handle hardware failures, network configuration, and physical maintenance. For most software-focused AI companies, building a data center operations team is a distraction from their core product roadmap.

The GPU Cloud Advantage

GPU cloud providers, particularly specialized platforms, offer a compelling alternative to on-premise hardware. By utilizing a cloud model, you shift the financial burden from capital expenditure to operational expenditure. You gain access to the latest generation of accelerators without the risk of hardware obsolescence.

More importantly, specialized GPU clouds provide the exact same level of data isolation and security as an on-premise deployment, provided they operate on sovereign infrastructure. You achieve the unit economics of highly optimized dedicated hardware without the multi-million dollar upfront investment. For teams projecting their infrastructure needs into the future, the flexibility to upgrade instance types instantly makes the GPU cloud model financially superior to locking into static on-premise hardware.

Ultimately, the break-even point for purchasing on-premise hardware versus renting cloud GPUs requires years of sustained, maximum utilization to justify the initial outlay. For the vast majority of AI scale-ups, the agility provided by specialized cloud infrastructure far outweighs the theoretical long-term savings of owning the metal.

The landscape of large language model inference is evolving rapidly, and the cost per token is expected to continue its downward trajectory. Understanding these future trends is essential for engineering leaders who are architecting systems that must scale efficiently over the next several years.

Next-Generation Hardware Efficiencies

The introduction of newer silicon, such as the H200 and B200 architectures, is already altering the unit economics of inference. These new chips offer significantly higher memory bandwidth and larger VRAM capacities. Because memory bandwidth is the primary bottleneck for generative AI inference, these hardware improvements will allow teams to serve much larger batch sizes simultaneously.

As throughput increases on these new architectures, the effective cost per token on dedicated infrastructure will plummet. A model that previously required multiple GPUs for tensor parallelism might soon run comfortably on a single instance, cutting infrastructure costs in half. Specialized providers are positioned to deploy these new architectures rapidly, passing the efficiency gains directly to users.

Software Optimization and Quantization

Beyond hardware, software optimizations are driving massive reductions in inference costs. Techniques like FP8 quantization, continuous batching, and speculative decoding are becoming standard practice. These methods reduce the memory footprint of large language models, allowing them to run faster and cheaper without a noticeable degradation in output quality.

As the open-source community continues to refine inference engines, the gap between expensive proprietary APIs and self-hosted open-source models will widen. Teams that invest in the engineering capability to manage dedicated infrastructure will capture the full financial benefit of these software advancements. The future of AI infrastructure belongs to organizations that can dynamically balance their workloads across highly optimized, dedicated GPU instances while maintaining strict control over their data sovereignty.

By staying ahead of these hardware and software curves, companies can ensure their AI products remain profitable even as user demand scales exponentially.

Sources

[1] NVIDIA: Rethinking AI TCO, Why Cost per Token Is the Metric That Matters; [2] AWS Documentation: Inference Options in Amazon SageMaker AI; [3] arXiv: A Cost-Benefit Analysis of On-Premise Large Language Model Deployment, Breaking Even with Commercial LLM Services; [4] Amazon EC2 On-Demand Pricing: p5.48xlarge, US East (N. Virginia), Linux, $55.04 per hour (read 3 August 2026)