Skip to content
MiraiTech
AI Lab

Experiment · Vision

Document vision parser

Vision models read the layout of a document and return the fields as JSON, with a confidence score and bounding box for each one. It handles receipts, invoices, tax forms and business cards without a template per format.

Type
Vision
Status
Live
Models
GPT-4o Vision, Claude Vision

Results

  • 96% field accuracy
  • 1.8 s per document
  • No templates to maintain

Field-level F1 was 96% across 500 documents in 8 categories. Processing took 1.8s and cost $0.03 per document on average, with no templates to maintain. Confidence scoring flagged 89% of extraction errors for human review.

What we tested

Traditional OCR needs a template for every document type. This experiment uses vision LLMs that read the layout and content directly instead. You upload a receipt, invoice or tax form and get structured JSON with field labels, values, confidence scores and bounding box coordinates, without defining a template.

Method

We tested GPT-4o Vision and Claude Vision on 500 real-world documents in 8 categories: receipts (120), invoices (100), tax forms (80), business cards (60), bank statements (50), purchase orders (40), contracts (30) and shipping labels (20). Field names, values and bounding boxes were annotated by hand as ground truth. AWS Textract and Google Document AI served as baselines. We measured field-level F1 and tracked processing time and cost per document.

Findings

  1. 01

    96% field accuracy with no templates

    GPT-4o Vision reached 96% field-level F1 across all document types with no template configuration. It matched Textract on receipts and did better on unusual layouts.

  2. 02

    Rotated and skewed documents still work

    The vision models stayed above 90% accuracy on documents photographed at angles up to 30 degrees. On the same rotated inputs, Textract dropped to 67%.

  3. 03

    Output schemas reduce hallucinated fields

    Pydantic response schemas with field-level validation cut hallucinated fields from 8% to 1.2%. The model invents less when it has to fill a fixed structure.

  4. 04

    Cost breaks even at about 50 document types

    At $0.03 per document, the vision approach costs 2x Textract per page, but there are no templates to build. Past about 50 document types, templates cost more to maintain.

How it is built

The FastAPI backend preprocesses each uploaded image (orientation detection, contrast normalization) and sends it to the vision model with an extraction prompt that specifies the output schema. The response is validated against the Pydantic schema. Confidence scores come from running the extraction twice and comparing which fields agree. Bounding box coordinates come from a second vision prompt focused on locating each field.

GPT-4o Vision and Claude Vision read the documents. FastAPI serves the extraction API and processes batch uploads asynchronously. Pydantic enforces the output schemas and validates extracted fields. A React frontend uses the Canvas API to draw bounding boxes over the original document for checking.

  • GPT-4o Vision
  • FastAPI
  • Pydantic
  • React
  • Canvas API

Problems we hit

Bounding box precision

Vision models describe roughly where things are instead of returning pixel coordinates. A second pass crops each predicted region and runs a focused extraction, which brings the boxes to within 5px.

Multi-page documents

Invoices of 3+ pages exceeded single-image context limits. We extract page by page, then a merge step removes duplicate fields and resolves references across pages, such as line items "continued on next page".