How to Estimate Serverless Inference Costs Before You Commit

AI This article was created with the help of AI.

The Core Identity: Converting Throughput to Cost

Engineering teams evaluating serverless inference APIs or reserved GPU capacity frequently face a financial comparison problem: provider rate cards express costs in two incompatible metrics. Managed serverless endpoints bill per million tokens (input versus output), whereas dedicated GPU infrastructure bills per instance hour. Comparing a $3.59 per hour GPU instance against a $0.60 per million output token API requires converting execution speed into financial performance. Without a rigorous mathematical identity connecting hardware throughput to token pricing, financial projections devolve into vendor-supplied estimates that collapse under real production workloads.

From GPU Hour to Cost Per Million Tokens

The direct relationship between hourly compute expenses and token generation cost is governed by sustained generation throughput. To convert an hourly GPU rate into a cost per million tokens, you must measure aggregate output tokens generated per second across the active cluster. The core mathematical identity is expressed as:

Cost Per 1M Tokens = (GPU Hourly Rate / Tokens Per Second / 3600) * 1,000,000

In this formula, the GPU hourly rate represents the total cost per hour for the reserved hardware instance, and tokens per second represents the total aggregate token generation rate emitted by the serving engine. Dividing the hourly rate by aggregate throughput yields the compute cost per individual token per hour. Multiplying by 1,000,000 scales the figure to a standardized million-token metric, while dividing by 3,600 converts the hourly instance rate into a per-second baseline. Simplifying the constant terms yields 277.78 * (Hourly Rate / Tokens Per Second).

Worked Example, Step by Step

Consider an engineering deployment running an open-weights model on a dedicated NVIDIA H100 GPU instance. The rate used below, $3.59 per GPU hour, is illustrative rather than a quote: it sits just under the $3.63 per hour average on-demand H100 price tracked across 49 cloud providers, where listings run from $0.45 per hour on spot up to $6.98 per hour on hyperscalers. To establish the baseline cost identity, evaluate three distinct serving throughput scenarios under steady-state execution:

The table is pure arithmetic from the identity above, applied to the illustrative hourly rate. Read down the rows: as aggregate throughput rises, the seconds of GPU time needed to emit a million output tokens fall in exact proportion, and the cost per million tokens falls by the same factor. The point is not the specific figures but the shape of the relationship: hardware hourly rates alone do not determine inference economics, because aggregate throughput is the primary cost driver. Published methodologies that use this identity note the same caveat, that the headline number assumes fully utilised hardware and must be divided by sustained average utilisation to be realistic. Treating throughput as a constant in a financial model therefore introduces severe estimation errors.

Where Throughput Numbers Come From

A common pitfall in inference cost forecasting is treating benchmark throughput numbers as fixed hardware specifications. A single NVIDIA H100 GPU does not possess a static tokens per second speed. Hardware throughput is an emergent property derived from the interplay between model parameter count, tensor precision, memory bandwidth saturation, context window length, request concurrency, and serving framework execution efficiency: published cost-per-token tables carry exactly that disclaimer, noting that actual throughput varies with model size, quantization, batch size and serving-engine configuration.

Why Throughput Is Not a Constant

Large language model inference consists of two distinct computational phases: prefill and decode. Autoregressive token generation during the decode phase is fundamentally bound by GPU memory bandwidth rather than floating-point computation capacity. For a 70-billion parameter model running in FP16 precision, every generated token requires loading 140 gigabytes of model weights from high-bandwidth memory (HBM3) into Tensor Cores. On an NVIDIA H100 SXM5 GPU with 3.35 terabytes per second of memory bandwidth, isolated single-sequence generation is physically capped at roughly 24 tokens per second per stream. Achieving hundreds or thousands of tokens per second requires running multiple concurrent requests in parallel to reuse loaded weights across batch sequences.

What to Measure Before You Estimate

Before accepting a vendor throughput claim or running cost projections, you must define the exact benchmarking environment. Synthetic benchmarks published in documentation frequently reflect ideal conditions: maximum batch size, fully saturated tensor parallelism, minimal prompt context, and unconstrained latency budgets. To establish a realistic throughput baseline, measure performance under five specific operational variables: target concurrency, average prompt length, expected output generation length, target time-to-first-token (TTFT), and inter-token latency limits.

Measuring throughput without bounding latency metrics yields invalid financial projections. Aggregate throughput keeps climbing as you raise batch size, but queuing time per request climbs with it: measured benchmarks on a single 70B FP8 deployment show cost per token at batch size 1 running 35x to 45x higher than at batch sizes of 64 to 128 with continuous batching, while a P95 time-to-first-token target below 500 ms caps the usable batch at roughly 8 to 16 sequences and a 2-second target opens it to 64 to 128. Cost models must evaluate throughput strictly within the latency constraints required by your end-user application, and treat any published benchmark as evidence gathered under stated conditions rather than a guarantee.

Lever One: Continuous Batching

In early LLM serving architectures, request batching relied on static, sequential execution. Incoming user prompts were grouped into fixed-size batches and processed together through the transformer layers until every sequence in the batch reached its generation limit.

The Limits of Sequential Serving

Static sequential batching creates severe GPU resource underutilization due to output length variance. Because different requests require different output lengths, shorter sequences complete early and emit end-of-sequence tokens while longer sequences continue generating. In static batching, early-terminating sequence slots remain idle, wasting VRAM allocation and compute cycles while waiting for the longest request to complete. This padding waste reduces aggregate throughput and inflates the effective unit cost per token generated.

Achieving Higher Throughput

Modern inference engines overcome static padding bottlenecks through iteration-level scheduling, widely known as continuous batching. First introduced at scale in serving frameworks like vLLM, continuous batching operates at the token iteration level rather than the request level. As soon as a sequence in a batch finishes generation, its allocated memory is freed, and a new request from the waiting queue is immediately inserted into the active execution step.

• Iteration-level scheduling eliminates idle VRAM slots by replacing finished requests on the subsequent forward pass.
• PagedAttention manages Key-Value cache memory in dynamic blocks, preventing memory fragmentation and enabling higher concurrency.
• Chunked prefill interleaves compute-heavy prompt processing with memory-bound decode steps to stabilize token generation latency.

By keeping GPU execution units saturated with active token generation, continuous batching dramatically increases aggregate throughput without requiring additional hardware or increasing hourly instance fees. Empirical benchmarks demonstrate that continuous batching can increase total LLM inference throughput by up to 23x compared to naive sequential execution while maintaining acceptable p50 latency. In our analysis of strategies, maximizing iteration scheduling density is the single most effective software optimization for lowering unit token costs.

Lever Two: Quantization and Precision

Quantization reduces the numerical precision of model weights and activations, directly compressing the VRAM footprint required to host large language models. Reducing precision from 16-bit floating-point (FP16 or BF16) to 8-bit or 4-bit representations fundamentally alters the memory bandwidth denominator in the cost identity.

The FP8 Advantage

NVIDIA Hopper architectures feature dedicated FP8 Tensor Cores, and on a per-SM basis Hopper's fourth-generation Tensor Cores deliver four times the matrix-multiply rate of A100 using the FP8 data type compared with previous-generation 16-bit floating point. In serving terms, vLLM reports that FP8 quantization allows a 2x reduction in model memory requirements and up to a 1.6x throughput improvement with minimal impact on accuracy. Halving the weight footprint is what lets a large model fit on a single 80 GB GPU rather than requiring a multi-GPU tensor parallel cluster, which removes hourly hardware from the numerator of the cost identity.

Trading Accuracy for Speed

While FP8 quantization preserves model accuracy across general reasoning tasks, moving to aggressive 4-bit schemes (such as AWQ or GPTQ) introduces measurable quality degradation. Published cost frameworks put the FP8 quality cost at roughly 0.3 to 0.5 benchmark points, while INT4 or AWQ deployments give up around 1.6 points, and they recommend evaluating quality per task before deploying. Lower precision formats can cause perplexity spikes in domain-specific tasks, mathematical reasoning, and structured JSON output generation. Engineering teams should never select precision based on theoretical hardware speedup alone: quantization decisions must be validated against your team's own evaluation harness to verify that token cost reductions do not compromise end-user task performance.

Lever Three: Prompt Shape and the KV Cache

Inference workloads are not uniform. A workload that ingests long input documents to produce short summaries exhibits completely different execution economics than a code-generation assistant that accepts a short prompt and generates a long block of output, even if total daily token volume is identical.

The Cost of Context

The prefill phase (processing input tokens) is compute-bound, maximizing Tensor Core utilization by running matrix operations across all prompt tokens in parallel. In contrast, the decode phase (generating output tokens) is memory-bandwidth-bound, executing one token step at a time while reading model weights and Key-Value (KV) cache history from HBM. Both phases bill at the same GPU hourly rate, so the distinction matters mainly when sizing batch configuration and latency budgets: retrieval and document workloads with long system prompts carry heavy prefill compute per request, while short-prompt generation workloads are decode-dominated. Storing KV cache states for long context windows also consumes substantial GPU memory, which restricts the maximum number of concurrent batch slots available, forcing lower serving density and raising effective token costs.

Benchmarking Production Shapes

Because prefill compute and decode memory consumption follow different scaling curves, serverless inference rate cards frequently price input and output tokens separately. Across published frontier-model list prices, output tokens are typically 3x to 5x the input price, reflecting the extended GPU occupancy required during generation.

To build an accurate cost estimate, analyze historic telemetry or synthetic production samples to calculate your application's exact input-to-output token ratio. Multiplying aggregate token estimates by generic average pricing without accounting for prompt shape leads to severe budget variance.

Lever Four: The 40 Percent Utilisation Reality

Most significant financial error in GPU cost forecasting occurs when engineering teams assume 100% continuous hardware utilization, which is exactly the assumption baked into the standard cost-per-token identity: a GPU is a fixed-cost asset whose hourly rate is identical at 10% load and at full load, so idle capacity spreads that charge across fewer delivered tokens. Calculating monthly costs by assuming a GPU cluster generates peak tokens every second for 730 hours per month results in aggressive cost underestimation.

The Danger of Perfect Utilisation

User demand for AI services is highly volatile. Enterprise applications experience predictable diurnal patterns, with traffic peaking during business hours and dropping significantly overnight and during weekends. To maintain low latency during peak traffic spikes, infrastructure teams must provision GPU capacity for peak load rather than average load. When traffic drops, reserved instances continue incurring hourly rental costs while sitting idle.

The 2.5x Cost Multiplier

Real-world production data tells a different story: published cost methodologies put actual GPU utilization in the 30% to 70% band over a billing cycle, well short of the fully saturated hardware the identity assumes. Take a working assumption near the low end of that band, roughly 40 percent average utilisation on an H100 instance: most of the hourly rental fee is then paying for idle clock cycles rather than generated tokens.

• Diurnal traffic fluctuations leave provisioned GPUs idle during off-peak hours.
• Headroom buffer provisioning required for sudden traffic bursts creates standing excess capacity.
• Slow cold-start latencies prevent traditional cloud auto-scalers from turning off idle instances quickly.
• Unbatched request arrivals during low-traffic windows prevent efficient iteration-level scheduling.

Operating at 40% average utilization introduces roughly a 2.5x cost multiplier onto nominal token generation math: a measured H200 deployment costs about $0.47 per million tokens at full saturation but about $1.17 per million once average utilisation drops to 40%. On the same arithmetic, a deployment estimated at $1.00 per million tokens under 100% utilization actually incurs an effective cost near $2.50 per million tokens once idle capacity is accounted for. Calculating your true GPU idle cost is essential before deciding whether to commit to fixed hourly compute instances.

Sanity-Checking Quotes and Per-Second Billing

When auditing vendor quotes or evaluating self-hosted instance economics, combine all four levers into a systematic verification workflow. Compare provider pay-per-token API estimates against reserved infrastructure proposals by evaluating total cost of ownership rather than raw hourly rates. Reviewing frameworks for cost per million tokens and analyzing pay-per-token versus dedicated deployments ensures your team selects the optimal commercial model.

Auditing the Denominator

To sanity-check a vendor proposal, audit the throughput assumptions embedded in their calculations. Verify whether their quoted cost per token relies on fully saturated batch utilization, FP8 precision, or synthetic prompt shapes that do not match your production telemetry. If a quote assumes near-perfect hardware utilization on bursty user traffic without offering automated scale-to-zero capabilities, the financial estimate is unviable.

Calculating Your True Savings

Modern cloud execution models address GPU waste by altering the underlying billing denominator. Rather than forcing teams to rent static instances by the hour, modern platforms implement per-second billing and serverless execution layers. At Lyceum, our infrastructure platform predicts job runtime, memory footprint, and GPU utilization before execution, allowing engineering teams to run serverless inference and compute workloads with per-second billing accuracy and zero idle charges.

Before signing long-term capacity commitments, benchmark your application on real hardware using an open serving framework such as vLLM, following its published deployment and benchmarking guidance. Test your actual prompt distributions, measure sustained tokens per second under production latency constraints, and check current rates against a live pricing page rather than a stale quote. Grounding your financial models in empirical throughput metrics prevents costly infrastructure miscalculations.