Experiment · Vision
Document vision parser
Vision models read the layout of a document and return the fields as JSON, with a confidence score and bounding box for each one. It handles receipts, invoices, tax forms and business cards without a template per format.
- Type
- Vision
- Status
- Live
- Models
- GPT-4o Vision, Claude Vision
Results
- 96% field accuracy
- 1.8 s per document
- No templates to maintain
Field-level F1 was 96% across 500 documents in 8 categories. Processing took 1.8s and cost $0.03 per document on average, with no templates to maintain. Confidence scoring flagged 89% of extraction errors for human review.
What we tested
Traditional OCR needs a template for every document type. This experiment uses vision LLMs that read the layout and content directly instead. You upload a receipt, invoice or tax form and get structured JSON with field labels, values, confidence scores and bounding box coordinates, without defining a template.
Method
We tested GPT-4o Vision and Claude Vision on 500 real-world documents in 8 categories: receipts (120), invoices (100), tax forms (80), business cards (60), bank statements (50), purchase orders (40), contracts (30) and shipping labels (20). Field names, values and bounding boxes were annotated by hand as ground truth. AWS Textract and Google Document AI served as baselines. We measured field-level F1 and tracked processing time and cost per document.
Findings
- 01
96% field accuracy with no templates
GPT-4o Vision reached 96% field-level F1 across all document types with no template configuration. It matched Textract on receipts and did better on unusual layouts.
- 02
Rotated and skewed documents still work
The vision models stayed above 90% accuracy on documents photographed at angles up to 30 degrees. On the same rotated inputs, Textract dropped to 67%.
- 03
Output schemas reduce hallucinated fields
Pydantic response schemas with field-level validation cut hallucinated fields from 8% to 1.2%. The model invents less when it has to fill a fixed structure.
- 04
Cost breaks even at about 50 document types
At $0.03 per document, the vision approach costs 2x Textract per page, but there are no templates to build. Past about 50 document types, templates cost more to maintain.
How it is built
The FastAPI backend preprocesses each uploaded image (orientation detection, contrast normalization) and sends it to the vision model with an extraction prompt that specifies the output schema. The response is validated against the Pydantic schema. Confidence scores come from running the extraction twice and comparing which fields agree. Bounding box coordinates come from a second vision prompt focused on locating each field.
GPT-4o Vision and Claude Vision read the documents. FastAPI serves the extraction API and processes batch uploads asynchronously. Pydantic enforces the output schemas and validates extracted fields. A React frontend uses the Canvas API to draw bounding boxes over the original document for checking.
- GPT-4o Vision
- FastAPI
- Pydantic
- React
- Canvas API
Problems we hit
Bounding box precision
Vision models describe roughly where things are instead of returning pixel coordinates. A second pass crops each predicted region and runs a focused extraction, which brings the boxes to within 5px.
Multi-page documents
Invoices of 3+ pages exceeded single-image context limits. We extract page by page, then a merge step removes duplicate fields and resolves references across pages, such as line items "continued on next page".