Best Open Model API for OCR and Document Extraction

AI This article was created with the help of AI.

What 'OCR' means once a vision-language model is doing it

Traditional document extraction pipelines treat optical character recognition as a detached two-step problem. First, an OCR engine scans pixel grids to detect bounding boxes, guess character coordinates, and output an unstructured string of text. Second, a separate script (often built from brittle regular expressions, spatial heuristics, or a downstream text-only parser) attempts to reconstruct tables, link key-value pairs, and coerce that raw text into a valid schema. When a document deviates slightly from the expected layout, such as an invoice with multi-line item descriptions or a scanned contract with misaligned columns, the heuristics fail and field extraction breaks.

Modern document extraction using vision language models collapses this fragmented process into a unified forward pass. Instead of decoupling character recognition from semantic understanding, a vision-language model ingests raw document images alongside a schema prompt. LlamaIndex describes the mechanism plainly: the image is processed through a vision encoder that captures spatial layout, typography, and visual structure, and that encoded representation is combined with the model's language understanding capabilities to produce a joint embedding, with results returned as structured JSON objects, key-value pairs, or markdown tables. The model understands that a numerical value sitting below a 'Total Amount Due' label belongs to that specific field without requiring manual bounding-box calculations.

• Legacy OCR pipelines: Image preprocessing -> bounding box detection -> text transcription -> regex/heuristic parsing -> schema mapping (5 distinct failure points).
• Vision-language extraction: Document image + JSON schema -> single transformer inference pass -> structured application-ready JSON (1 unified step).
• Layout resilience: Vision-language models interpret non-standard grids, rotated scans, embedded tables, and handwritten annotations in their native spatial context.

By eliminating intermediate regex parsing stages, engineering teams reduce maintenance overhead and avoid building custom template parsers for every new vendor invoice or contract format.

Why document extraction is the job where hosting region stops being abstract

In many machine learning applications, compute location is treated as an abstract networking metric evaluated only by round-trip latency. Document extraction is the workload where hosting location becomes an immediate legal boundary. Business documents submitted to an extraction API (invoices, employment agreements, purchase orders, medical reimbursement forms, tax filings, and identity scans) almost always carry direct personal identifiers and commercial data.

Under the General Data Protection Regulation (GDPR), transmitting unstructured documents containing personal data to third-party endpoints located outside the European Economic Area introduces regulatory friction. When an API endpoint routes document images through US-hosted infrastructure, the transfer falls under the jurisdictional scope of foreign surveillance frameworks such as the US CLOUD Act, complicating compliance under GDPR Chapter V. For European teams processing corporate documents at scale, data residency is not a marketing preference; it is a foundational infrastructure requirement.

• Direct personal data exposure: Documents contain names, IBANs, tax numbers, home addresses, and confidential contract terms that cannot be stripped prior to OCR without defeating the extraction task.
• Jurisdictional compliance: Routing document payloads through sovereign European data centers eliminates complex international transfer mechanisms and cross-border regulatory exposure.
• Audit transparency: Retaining document processing strictly within EU borders simplifies vendor risk assessments and data protection impact assessments (DPIAs).

The two vision-language models available, and what each costs

There is no proprietary, black-box OCR product here. Instead, Serverless Inference provides access to leading open-weight multimodal models running on sovereign European infrastructure in the eu-north1 region. Document extraction is performed directly through these vision-language models via an OpenAI-compatible endpoint.

Two models in the serverless catalogue serve document extraction workloads: Qwen2.5-VL-72B and MiniCPM-V 4.5. Both models accept document images directly in chat completion requests, run in the eu-north1 region with guaranteed European data residency, and bill strictly per token with zero base platform fees.

Both models operate under a pay-per-token model without upfront cluster commitments or idle hardware costs. Because they are exposed through standard OpenAI-compatible API paths, switching an existing extraction pipeline to either model requires updating only the base URL and the model parameter string.

Why the cheaper model here is the larger one

In most cloud model catalogues, pricing scales directly with parameter count: larger dense models cost significantly more per token than lightweight edge architectures. In the serverless multimodal catalogue, this relationship is inverted. Qwen2.5-VL-72B, which sits in the Standard tier as a high-capacity model, is priced at $0.25 per 1M input tokens and $0.75 per 1M output tokens. Meanwhile, MiniCPM-V 4.5, located in the Fast tier, costs $0.66 per 1M input tokens and $1.11 per 1M output tokens.

This pricing structure means engineering teams do not face an economic penalty when selecting the larger 72B parameter architecture for high-density document parsing. When extracting deeply nested tables, fine print, multi-column balance sheets, or dense legal clauses, developers can default to Qwen2.5-VL-72B without inflating token expenditure.

• High-capacity architecture at lower cost: Qwen2.5-VL-72B provides substantial parameter capacity for intricate layouts at $0.25/$0.75 per million tokens.
• Tier dynamics: The Fast tier classification for MiniCPM-V 4.5 reflects inference throughput characteristics rather than a per-token price ceiling.
• Workload selection: Teams processing high volumes of complex PDF pages can utilize the 72B model directly, reserving smaller architectures for workloads where specific inference latency profiles are required.

Understanding this dynamic enables FinOps leads and ML engineers to optimize both extraction fidelity and infrastructure spend simultaneously across large-scale document pipelines.

What zero data retention does and does not cover

When processing sensitive financial and legal records, data retention policies dictate system architecture. Serverless Inference runs on a self-asserted zero-data-retention policy. Prompts, document images, and extracted completions are processed transiently in volatile GPU memory and are never persisted to a database, log store, or long-term disk volume.

It is critical for engineering teams to understand the technical boundary of this guarantee. Zero data retention is self-asserted, with no third-party compliance attestations behind it. Prompts and visual tokens are cached in GPU memory (such as within framework-level KV caches) strictly for the duration of the active inference session (typically a few minutes at most) to handle generation. Once the token stream completes and the connection closes, the allocated memory buffers are recycled. Data never touches permanent storage or retraining datasets.

• Zero disk persistence: Ingested images and generated JSON payloads are never written to disk or stored in long-term databases.
• No model retraining: Customer payloads are never logged or repurposed for model fine-tuning or weight training.
• Transient GPU memory caching: Context tokens reside in volatile GPU memory only for active generation and short session windows before deallocation.
• No SLA on serverless: Serverless Inference is a self-serve, pay-per-token endpoint without contractual availability tiers, uptime numbers, or service credits. Platform operational health is monitored via status.lyceum.technology.
• Published Data Processing Agreement: A standard Data Processing Agreement (DPA) governing data controller and processor relationships is available on the website.

Teams needing contractual service-level agreements or dedicated compute isolation can deploy dedicated GPU VMs or private inference endpoints, while serverless pipelines benefit from immediate per-token billing under the self-asserted zero-retention policy.

Building the extraction call: structured output from a document

Because the inference engine implements an OpenAI-compatible API interface, integrating open vision-language models into an existing application requires minimal code modification. Rather than asking the model to return freeform text and attempting post-hoc parsing, developers can pass a JSON schema (used directly, or extracted from a Pydantic model) in the request's response_format parameter. The vLLM documentation shows exactly this pattern for constrained generation on an OpenAI-compatible server, with xgrammar or guidance enforcing the schema during decoding.

Using schema-guided decoding or response format parameters ensures that the model outputs strictly valid JSON matching your application's data model. The inference engine constrains token generation at the sampler level, preventing missing fields, unexpected markdown wrappers, and malformed syntax.

1. Encode the target document page as a standard base64 image string or provide an accessible URL.
2. Define a strict JSON schema or Pydantic data model specifying required fields (such as invoice_number, vendor_name, line_items, tax_amount, and total_due).
3. Pass the schema into the response_format parameter of the chat completion call.
4. Receive the validated JSON object directly in the response payload without secondary parsing stages.

The following implementation demonstrates how to execute a structured invoice extraction call against the Serverless Inference endpoint using the standard OpenAI Python client:

``python import base64 from openai import OpenAI client = OpenAI( base_url="https://api.lyceum.technology/api/v2/external/serverless", api_key="YOUR_LYCEUM_API_KEY", ) def encode_image(image_path: str) -> str: with open(image_path, "rb") as f: return base64.b64encode(f.read()).decode("utf-8") image_b64 = encode_image("invoice_scan.png") invoice_schema = { "name": "invoice_extraction", "strict": True, "schema": { "type": "object", "properties": { "invoice_id": {"type": "string"}, "issue_date": {"type": "string"}, "vendor": {"type": "string"}, "total_amount": {"type": "number"}, "line_items": { "type": "array", "items": { "type": "object", "properties": { "description": {"type": "string"}, "quantity": {"type": "number"}, "price": {"type": "number"} }, "required": ["description", "quantity", "price"], "additionalProperties": False } } }, "required": ["invoice_id", "issue_date", "vendor", "total_amount", "line_items"], "additionalProperties": False } } response = client.chat.completions.create( model="Qwen/Qwen2.5-VL-72B-Instruct", messages=[ { "role": "user", "content": [ {"type": "text", "text": "Extract the invoice data according to the supplied schema."}, {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image_b64}"}} ] } ], response_format={"type": "json_schema", "json_schema": invoice_schema} ) print(response.choices[0].message.content) ``

How to measure whether it is accurate enough for your workflow

Generic public benchmarks do not reflect the idiosyncrasies of proprietary business documents. The PureDocBench paper, which audits OmniDocBench (the dataset most document-parsing models are still scored on), reports that its nine categories miss high-frequency enterprise types such as financial invoices, medical records, legal contracts, and logistics documents, and that it contains no photographically captured or physically degraded pages, leaving real-world robustness untested; the same three-stage audit screened 21,353 evaluator-scored blocks and confirmed 2,580 errors, an error rate of 12.08%. Production pipelines, by contrast, handle low-resolution faxes, skewed smartphone photos, multi-currency receipts, and complex multi-page tables. Because the catalogue publishes no character error rates or synthetic accuracy rankings for these models, engineering teams should establish empirical evaluation harnesses on their own document corpora.

To determine whether multimodal AI inference meets your quality threshold, build a representative test set of 50 to 100 real-world documents spanning your edge cases. Measure extraction precision at the field level rather than evaluating pure string similarity across the whole page.

• Field-level exact match: Compare parsed values (such as currency amounts, dates, and identification numbers) against human-annotated ground truth.
• Table reconstruction fidelity: Verify that row-column alignments, item quantities, and unit calculations match across multi-line order tables without dropped rows.
• Schema adherence rate: Track the percentage of API calls that return schema-valid JSON without null fields or format violations.
• Cost per document: Calculate the total token consumption (input image tokens plus generated JSON tokens) to establish accurate unit economics per document type.

Run a page of your own documents through both Serverless Inference models (Qwen2.5-VL-72B and MiniCPM-V 4.5) and compare the extracted fields against your operational baseline to select the right price-to-performance fit.