The Anatomy of RAG Compute

A production Retrieval-Augmented Generation (RAG) pipeline operates across three distinct compute domains. Treating them as a single workload is a common architectural mistake. When building a system that scales, you must understand that each phase has entirely different hardware requirements.

Ingestion and Embedding Workloads

The first stage converts raw text, documents, and data into vector representations. This process requires high-throughput, batch-optimized compute. Embedding models are generally smaller than generation models but process massive volumes of text during initial ingestion and continuous updates. Using expensive hardware designed for generation to run embedding tasks is highly inefficient.

Retrieval and Reranking Infrastructure

Searching the vector database and scoring candidates forms the second domain. This stage is typically memory-bandwidth bound. For many applications, this relies on CPU clusters. However, as accuracy requirements increase, teams often implement a two-stage retrieval process. This involves a fast initial search followed by a cross-encoder reranker. Rerankers require small GPUs to process the semantic relationship between the query and the retrieved chunks.

Generation and LLM Compute

The final LLM call is heavily compute-bound and VRAM-intensive. This is where the actual text generation occurs, especially as retrieved context windows expand. The generation phase requires the most significant hardware investment.

Measuring Total Inference Cost

Google Cloud's GKE Inference Quickstart documentation [3] prices each serving profile in dollars per million input tokens and dollars per million output tokens, which is the granularity this needs. Teams must measure total inference cost across all these steps rather than solely looking at the main LLM token bill. When you isolate these components, you can provision the exact hardware required for each task. This prevents a scenario where an expensive H100 sits idle waiting for a CPU-bound vector search to complete. By tracking the cost per query across the entire pipeline, organizations can identify bottlenecks and optimize their infrastructure spend effectively. Proper isolation ensures that your RAG pipeline remains both performant and cost-efficient under heavy load.

VRAM Math and the KV Cache Trap

When sizing GPUs for standard LLM inference, engineers typically calculate the VRAM required for model weights. A standard rule of thumb is 4 bytes per parameter for FP32, 2 bytes for FP16, 1 byte for FP8, and 0.5 bytes for INT4. For example, a 70B parameter model in INT4 requires about 35GB of VRAM just to load the weights into memory, because 70 billion parameters at half a byte each is 35GB.

The Impact of Context Windows

RAG fundamentally breaks this standard math. Because RAG relies on injecting retrieved documents into the prompt, you are dealing with massive context windows. As context scales to 32K, 128K, or 256K tokens, the Key-Value (KV) cache becomes the dominant consumer of GPU memory. The KV cache stores intermediate attention calculations to prevent recomputing them for every new token generated.

If you retrieve 20 chunks of text and send 32,000 tokens to the LLM, the KV cache expands linearly. With large context windows, the activations and KV cache often exceed the memory needed for the model weights themselves. If you do not account for this expansion, your pipeline will crash with Out of Memory (OOM) errors during peak retrieval loads.

Effective Mitigation Strategies

To prevent these catastrophic failures, infrastructure teams must implement specific memory management techniques. First, implement PagedAttention, which is standard in frameworks like vLLM, to manage KV cache memory dynamically and reduce fragmentation.

Second, size the cache before you size the model. NVIDIA's analysis of RAG deployment [1] puts the KV cache for Llama-2-70B at around 40GB for a batch of 32 at a 4,096-token context, and that figure grows with both batch size and context length. Chunking strategies that prioritize density over length hold it down. A 512-token chunk with high semantic value is significantly better than a 2,048-token chunk filled with noise. Smaller, denser chunks reduce the total token count sent to the LLM, directly lowering KV cache pressure. Finally, enable chunked prefill to prevent long RAG prompts from stalling concurrent requests, ensuring your infrastructure maintains high throughput even when processing extensive retrieved contexts.

Self-Hosting Embeddings vs. Managed APIs

Choosing an embedding model dictates the baseline cost of your ingestion pipeline. While managed APIs offer a low barrier to entry for prototypes, self-hosting embedding models on dedicated GPU infrastructure becomes mathematically superior at scale.

The Cost Premium of Managed APIs

Managed embedding endpoints bill per million tokens, so the line item tracks the size of your corpus rather than the size of your traffic. A pipeline that processes 1 billion tokens a month is billed 1,000 times the per-million rate for that one component, and re-embedding the corpus after a model upgrade repeats the charge in full. The cost scales linearly with data volume, which is what makes continuous ingestion into a large knowledge base expensive.

Economics of Self-Hosted Models

Self-hosting models like BGE-M3 (1,024-dimension vectors, 8,192-token sequences) or Qwen3-Embedding-8B (up to 4,096 dimensions, 32K context) on a dedicated GPU turns a variable bill into a fixed one. The break-even is arithmetic rather than a rule of thumb: divide the monthly cost of the GPU by the per-million-token rate you would otherwise pay. An L40S on-demand VM on Lyceum lists at $1.19 per GPU-hour, so holding one for a 730-hour month costs about $869. Against a managed rate of $1.00 per million tokens that pays for itself at roughly 869 million tokens a month. Against $0.05 per million it takes about 17 billion. Per-token embedding rates are low enough that the crossover usually sits far higher than teams assume, so run that division on your own rate and your own volume, and read the current list on the pricing page, before you migrate. What survives the arithmetic is the non-price case: no rate limits, no surprise repricing, and the freedom to run a larger multilingual model.

Deploying on Sovereign Infrastructure

This is where infrastructure choices matter immensely. Lyceum provides raw GPU access via SSH from European data centres in Spain, Paris and the Nordics, billed per second with no base fee. Deploying a Hugging Face Text Embeddings Inference (TEI) container on a Lyceum virtual machine trades a per-token bill for a per-second one. TEI is highly optimized for batch processing, maximizing the throughput of your embedding workloads. TEI's own documentation lists token-based dynamic batching and Flash Attention among its features [2], which is where the throughput on your allocated hardware comes from. Furthermore, self-hosting keeps proprietary documents inside infrastructure you control. For organizations handling sensitive financial, medical, or legal documents, sending data to external APIs is often a strict compliance violation. Self-hosting on Lyceum keeps your ingestion pipeline under your own control at a predictable cost, and the processing stays in European data centres.

Vector Database Infrastructure: CPU or GPU?

The retrieval stage introduces an entirely different hardware question for engineering teams: do you need GPUs for your vector database? The answer depends heavily on the scale of your data and your latency requirements.

CPU-Based Vector Search

For datasets under 5 million vectors, CPU-based infrastructure running PostgreSQL with the pgvector extension or dedicated databases like Qdrant is often sufficient. Query latency here depends on index type, vector count and the recall target you tune for, so measure it on your own corpus rather than budgeting from a published band. CPU instances are significantly cheaper than GPU instances and scale horizontally with much less friction. For many internal knowledge bases or standard customer support bots, this level of performance is perfectly acceptable.

Scaling to GPU Acceleration

However, as your knowledge base scales into the tens or hundreds of millions of vectors, CPU search becomes a severe bottleneck. At this massive scale, GPU-accelerated vector databases become one of the few ways to hold retrieval latency flat as the index grows. GPUs excel at the highly parallel matrix multiplications required for exact or approximate nearest neighbor searches across massive datasets. When a user submits a query, the system must compare the query vector against millions of stored vectors simultaneously. GPUs handle this parallel workload far faster than CPUs.

The Role of Reranking Hardware

Beyond the initial search, modern RAG pipelines often implement a two-stage retrieval architecture. This uses fast, dense retrieval followed by a cross-encoder reranker. If you implement this pattern, you will need a small GPU specifically dedicated to the reranking model. Rerankers are transformer models that score the relevance of retrieved chunks against the query. While they drastically improve the accuracy of the final LLM generation, they require dedicated compute. A small inference GPU such as an NVIDIA L4 or an L40S is typically enough for this task, providing the necessary inference speed without the high cost of flagship generation GPUs.

Production Best Practices for RAG Infrastructure

Building a RAG pipeline that survives production traffic requires strict infrastructure discipline. Treating a RAG system like a simple API wrapper leads to performance degradation and cost overruns. Follow these architectural principles to ensure stability.

Decouple Your Compute Resources

Never run your embedding model, vector database, and generation LLM on the same machine. Isolate them so you can scale the generation GPU independently of the ingestion pipeline. Embedding tasks are often batch-heavy and run in the background, while generation tasks are user-facing and require immediate, low-latency responses. By decoupling these services, you prevent a massive document upload from starving your generation model of the compute it needs to answer user queries.

Implement Scale-to-Zero Economics

RAG workloads are often bursty. You might experience heavy traffic during business hours and near-zero traffic overnight. Paying for an idle H100 overnight destroys your unit economics. Utilize infrastructure that supports scale-to-zero, ensuring you only pay per second when serving traffic. This approach allows you to maintain high availability without bleeding budget on unused hardware.

Enforce Strict Data Sovereignty

For European enterprises, sending proprietary documents to non-EU APIs for embedding or generation raises compliance questions your data protection officer has to answer. Host your models in European data centres so processing stays inside the EU. Lyceum runs GPU compute in European data centres in Spain, Paris and the Nordics, billed per second with no base fee, and provides an inference platform that lets teams run any LLM or embedding model on it. With full OpenAI SDK compatibility, you can swap out managed APIs for dedicated endpoints with zero code changes, but this alone does not make the endpoints GDPR-compliant. You maintain open-stack transparency using vLLM and TensorRT-LLM, avoiding the vendor lock-in of proprietary black-box engines while securing your corporate data. Furthermore, by utilizing open-source serving engines, your engineering team retains full control over generation parameters, token limits, and system prompts. This level of control is essential for fine-tuning the RAG pipeline to meet strict enterprise performance standards.

Tracking and Controlling RAG Inference Costs

Managing a hardware budget for AI applications requires precise visibility into how resources are consumed. Without tracking, RAG pipelines can become a financial liability.

Identifying Cost Centers

As Google Cloud's inference cost guidance shows [3], controlling inference costs requires a granular understanding of your pipeline. You cannot look at the end-of-month cloud bill. You must isolate the cost of embedding generation, vector storage, reranking compute, and the final LLM generation. Often, teams discover that their retrieval mechanism or overly aggressive chunking strategy is driving up costs unnecessarily. By breaking down the pipeline, you can identify exactly which component is consuming the most resources.

Establishing Unit Economics

To build a sustainable application, engineering teams must establish clear unit economics. The most critical metric is the cost per query. This metric aggregates the compute time spent on embedding the user query, searching the database, reranking the results, and generating the final response. When you track the cost per query, you can make informed decisions about hardware provisioning. For instance, you might find that switching to a smaller, quantized LLM reduces the cost per query by half without significantly impacting the quality of the answers.

Continuous Optimization Strategies

Once you have visibility into your metrics, you can apply continuous optimization strategies. This might involve adjusting your chunk size, limiting the number of retrieved documents, or switching to a more efficient embedding model. Additionally, utilizing batch processing for non-urgent queries can drastically reduce compute overhead. By treating cost optimization as an ongoing engineering task rather than a one-time setup, you ensure that your RAG infrastructure remains viable as user adoption grows and data volumes expand. Furthermore, implementing robust caching mechanisms for frequently asked questions can bypass the entire compute pipeline entirely. If a user asks a common question, serving a cached response eliminates the need for embedding, retrieval, and generation, driving the cost per query for that specific interaction down to near zero.

Advanced Chunking Strategies for VRAM Efficiency

Data preparation directly impacts hardware requirements. In a RAG system, document chunking is not just a data processing step, it is a critical VRAM management strategy.

The Problem with Naive Chunking

Many early RAG implementations use naive chunking, splitting documents arbitrarily every 1,000 tokens. This approach creates significant inefficiencies. If a chunk contains only 100 tokens of relevant information and 900 tokens of irrelevant filler, you are forcing the LLM to process useless data. This wastes compute cycles and unnecessarily inflates the KV cache. As NVIDIA's RAG deployment analysis shows [1], KV cache size scales with both batch size and context length, so every unnecessary token sent to the model consumes GPU memory a concurrent request could have used.

Semantic and Density-Focused Chunking

To optimize infrastructure, teams must adopt semantic chunking strategies. This involves breaking documents down based on their actual meaning, such as splitting by paragraphs, sections, or logical concepts. A smaller, highly dense chunk of 256 tokens that perfectly answers a query is far superior to a massive chunk that dilutes the context. By prioritizing density, you reduce the total payload sent to the LLM. This directly lowers the KV cache requirements, allowing you to serve more concurrent users on the same GPU hardware without triggering memory errors.

Dynamic Context Assembly

Advanced pipelines take this a step further by implementing dynamic context assembly. Instead of hardcoding a fixed number of chunks to retrieve, the system evaluates the complexity of the query and retrieves only what is strictly necessary. If a query can be answered with a single dense chunk, the system sends a minimal prompt. If the query requires synthesizing information from multiple sources, the system expands the context window accordingly. This dynamic approach ensures that you only consume maximum VRAM when the task genuinely requires it, preserving your hardware budget for peak loads.

Architecting Compute for Cross-Encoder Rerankers

As RAG applications mature, higher accuracy requirements lead to reranking models. Understanding how to provision hardware for this specific component is crucial for maintaining low latency.

The Function of a Reranker

Standard vector search relies on dense embeddings to find similar documents. While fast, it often struggles with nuanced semantic relationships. A cross-encoder reranker solves this by taking the user query and a retrieved document, processing them together through a transformer model, and outputting a highly accurate relevance score. However, because cross-encoders process the query and document simultaneously, they are computationally heavy. You cannot run a high-performance reranker efficiently on a CPU.

Dedicated GPU Provisioning

Hugging Face's Text Embeddings Inference documentation lists cross-encoder rerankers such as BAAI/bge-reranker-large among the models it serves [4], on the same container and the same hardware classes as the embedding models. Unlike the massive VRAM requirements of generation LLMs, rerankers are relatively small models. They do not need an 80GB H100. Instead, they run well on smaller, cost-effective GPUs. In the wider market that means parts like the NVIDIA L4; on Lyceum the smallest storefront GPU is the L40S, listed at $1.19 per GPU-hour for an on-demand VM and $1.69 for dedicated inference or serverless training. By isolating the reranker on its own dedicated hardware, you prevent it from competing for resources with your embedding or generation models. This isolation is critical for maintaining predictable latency across the pipeline.

Optimizing Reranker Throughput

To maximize the efficiency of your reranking hardware, teams should utilize optimized serving frameworks like Text Embeddings Inference (TEI). TEI supports both embedding and reranking models, offering dynamic batching and optimized execution graphs. When deployed on Lyceum infrastructure, TEI allows your reranker to process dozens of document candidates in milliseconds. This ensures that the two-stage retrieval process does not introduce unacceptable delays into the user experience. By right-sizing the GPU for the reranker and using optimized software, you achieve state-of-the-art retrieval accuracy without breaking your infrastructure budget. Furthermore, because reranking workloads are highly parallel, you can easily scale these smaller GPU instances horizontally as your traffic grows. This modular approach to infrastructure design ensures that your RAG pipeline remains agile, allowing you to upgrade individual components, like swapping in a newer reranking model, without overhauling your entire hardware stack.

Sources

[1] NVIDIA Technical Blog: Deploying RAG Applications on NVIDIA GH200; [2] Hugging Face: Text Embeddings Inference Documentation; [3] Google Cloud: Analyze Model Serving Performance and Costs with GKE Inference Quickstart; [4] Hugging Face: Text Embeddings Inference, Supported Models and Hardware