How Receipt Parsing Works: Inside Yomio's OCR Pipeline (AWS + Azure)

When you scan a receipt in Yomio, a lot happens between pressing the shutter button and seeing the categorized transaction in your dashboard. Your receipt photo travels through a multi-stage OCR pipeline that extracts, normalizes, and enriches the raw text into structured financial data.

This article walks through what happens inside that pipeline. It's technical — but it's also useful for understanding why some receipts scan perfectly and others need corrections, and how we're working to close that gap.

Key Takeaways

• Yomio uses both AWS Textract and Azure Document Intelligence — dual-provider architecture for redundancy and accuracy
• Pipeline has 5 stages: ingest, OCR extraction, normalization, categorization, enrichment
• Line-item extraction is the hardest problem — receipts have no standard format
• Confidence scoring flags low-confidence fields for human review
• Multi-modal support — receives images, PDFs, and structured data
• Current accuracy: 94.7% line-item extraction, 99.3% total capture
• Processing time: 2–6 seconds per receipt from scan to categorized entry

Pipeline Overview

The OCR pipeline has five stages:

1. Ingest — image preprocessing and format validation
2. OCR Extraction — text and structure extraction via AWS Textract / Azure Document Intelligence
3. Normalization — merchant name matching, date formatting, currency detection
4. Categorization — line-item and total-level category assignment
5. Enrichment — barcode lookup, product data, user account linking

Each stage has built-in fallbacks. If a stage produces low-confidence output, the pipeline flags it rather than propagating bad data forward.

Stage 1: Ingest — Image Preprocessing

Before OCR engines touch the receipt, the image passes through preprocessing:

Format normalization:

• Photos are resized to a maximum of 2048px on the longest edge
• JPEG compression is applied (quality 85%) for storage efficiency
• PDF pages are rendered to images at 300 DPI

Quality checks:

• Blur detection — if the image is too blurred (Laplacian variance below threshold), the user is asked to retake
• Skew detection — receipts shot at extreme angles are corrected via perspective transform
• Contrast normalization — low-contrast receipts (faded thermal paper) get adaptive histogram equalization

Multi-page support:

• Receipts longer than one page are automatically detected (page break markers, folded edges)
• Multi-page receipts are stitched into a single document for processing

Stage 2: OCR Extraction — AWS Textract and Azure Document Intelligence

This is the core of the pipeline. Yomio uses two OCR providers:

AWS Textract:

• Primary provider for standard receipts
• Best-in-class for printed text extraction
• Receipt API (Expense) specifically trained on receipt layouts
• Detects: merchant name, transaction date, total, subtotal, tax, line items, payment method

Azure Document Intelligence (Form Recognizer):

• Secondary provider for receipts with complex layouts
• Better at detecting table structures within receipts
• Used as fallback when Textract confidence scores are below threshold
• Better handwriting detection for handwritten receipts

Provider selection logic:

1. Primary attempt: AWS Textract
2. If Textract confidence < 70% on key fields: also run Azure Document Intelligence
3. Compare outputs — pick the higher-confidence result for each field
4. If both below threshold: flag receipt for manual review

This dual-provider architecture means that if one provider has a blind spot for a specific receipt format (common with non-English receipts or unusual layouts), the other provider can compensate.

What Each Provider Extracts

Both providers extract similar fields, but with different strengths:

Stage 3: Normalization — Making Data Consistent

Raw OCR output is accurate but messy. "Starbucks Coffee #4521" and "Starbucks" need to be recognized as the same merchant. "05/08/2026" and "May 8, 2026" need to be the same date. This stage handles those transformations.

Merchant normalization:

• Fuzzy string matching against a merchant database (over 50,000 merchants)
• Brand-level grouping: "Starbucks Coffee #4521" → "Starbucks"
• Chain identification: local franchises mapped to their parent brand
• User-defined aliases: if you rename a merchant once, the system learns it

Date normalization:

• Detects date format from receipt (US: MM/DD/YYYY, EU: DD/MM/YYYY, ISO: YYYY-MM-DD)
• Resolves ambiguity using context (a receipt from a UK merchant on 03/04/2026 → April 3, 2026)
• Time zone detection from merchant location

Currency normalization:

• Currency symbol detection ($, €, £, ¥, etc.)
• Three-letter code detection (USD, EUR, GBP)
• Ambiguity resolution ($ could be USD, CAD, AUD, MXN — resolved via merchant location)

Amount validation:

• Cross-checks: subtotal + tax = total? Verified against line-item sum
• Discount detection: if line items sum minus total > threshold, a discount was applied
• Tip detection (restaurant receipts): difference between subtotal and total when tip line exists

Stage 4: Categorization — Intelligent Tagging

Extracted line items need to be assigned to budget categories. This happens through a two-layer system:

Layer 1: Merchant-based rules

• Known merchants have default category mappings
• "Kroger" → Groceries (default), "Shell" → Transportation (default), "Netflix" → Entertainment (default)
• Users can override per-merchant: "I want Shell categorized as Transportation"

Layer 2: Item-level AI categorization

• For unknown merchants or mixed-category receipts (Target with both groceries and electronics)
• Each line item is categorized separately based on product name, price, and historical data
• Categories include: Groceries, Dining, Transportation, Shopping, Entertainment, Healthcare, Utilities, Housing, Education, Personal Care, and 12+ subcategories

The categorization confidence score determines whether the assignment is automatic or suggested for review. At 90%+ confidence, the category is applied silently. Below that, it's highlighted for user confirmation.

Stage 5: Enrichment — Connecting to Your Account

The final stage connects the processed receipt to your Yomio account:

• User assignment — the receipt is linked to your account (or family group if sharing is enabled)
• Inventory update — line items matching known inventory items trigger quantity updates (see our inventory deep dive)
• Subscription detection — recurring merchants with regular amounts are flagged as potential subscriptions
• Barcode resolution — scanned barcodes are matched against Open Food Facts and product databases for enrichment
• Streak update — the daily streak counter is incremented
• XP award — scanning XP is credited to your account

Processing Time and Scalability

• Average processing time: 2–6 seconds per receipt
• Peak throughput: thousands of receipts per minute (Lambda-based auto-scaling)
• Storage: receipt images in S3, structured data in PostgreSQL
• CDN: receipt images served via CDN for fast retrieval

Accuracy Benchmarks

We continuously measure pipeline accuracy against a manually curated test set of 10,000 receipts:

Where errors still occur:

• Handwritten receipts (30% lower accuracy than printed)
• Severely damaged thermal paper (faded, curled, torn)
• Receipts with non-standard discount structures (BOGO, loyalty point redemptions)
• Micro-printed terms and conditions (OCR ignores these intentionally)

Future Improvements

The pipeline is actively being improved in three areas:

1. Handwriting recognition. Current handwriting accuracy is the biggest gap. We're training custom models on receipt-specific handwriting (tip amounts, handwritten merchant names, personal notes on receipts).

2. Receipt format coverage. International receipts have different layouts, tax structures, and languages. We're expanding the provider training data to cover more regional formats.

3. Real-time correction suggestions. Instead of requiring manual correction after scanning, the next pipeline version will suggest corrections during the scan flow — before the receipt is finalized.

FAQs