AI Policy PDF Annotation

Key results

• 4x — Manual VA workload replaced each month
• 24/7 — Automated annotation availability
• 99.4% — Peak annotation accuracy in validation set

TL;DR

• Built an AI pipeline that converts complex government benefit PDFs into machine-readable JSON with stable field-level coordinates and input metadata.
• The system uses PDF parsing, OCR-aware preprocessing, and multi-pass LLM validation to identify every input, its type, heading context, and checkbox default state.
• Accuracy is monitored with cross-model consensus logic to reduce hallucination risk and improve trust for LLM-driven form workflows.
• Result: a continuously running automation layer that saves the equivalent of 4 full-time assistants each month while processing policy forms at scale, 24/7.

Project Overview

We partnered with Ito Health to modernize how Rhode Island-focused policy documents become AI-ready inputs for LLM systems.

Policy PDFs are often inconsistent in layout, with mixed form structures, nested headings, and dense conditionals. Our goal was to transform these documents from brittle static files into structured, reliable input for automation and AI agents.

The solution first extracts text layers and layout coordinates using PDF parsing, then runs multiple model passes to classify each detected input field and infer semantics like labels, option sets, and dependency context. The output is normalized into machine-readable annotations with geometry.

• Detects input elements (text boxes, checkboxes, dates, signatures, dropdown-like fields) and emits a consistent schema
• Captures absolute XY location for each field to keep downstream OCR and UI automation aligned
• Classifies field types and maps each to the most likely section heading and policy context
• Adds checkbox state inference (checked/unchecked/default) with confidence and verification flags

To reduce hallucination, the pipeline is built with redundancy: independent extraction and interpretation passes, semantic reconciliation, and validation checks before an annotation is finalized.

Business Challenge / Problem Statement

Why policy PDFs were blocking reliable AI workflows.

The Rhode Island benefit documentation set for TANF, SNAP, Medicare, and Medicaid varies by update cycle and form revision, making deterministic extraction impossible with a single parser.

Manual review was expensive and slow, especially for variable layouts and checkbox-heavy forms where a wrong default interpretation can break downstream eligibility logic.

LLMs were useful for extraction but produced occasional hallucinations when uncertain, which is unacceptable for public-benefit systems that require high trust and consistent policy handling.

• Unstructured Inputs — Field layouts differ by program, making traditional rule-only extraction unreliable.
• High Cost of Human Review — Teams spent significant manual effort normalizing forms before AI systems could act.
• Hallucination Risk — Model-only extraction introduced uncertain outputs without strict, auditable validation.

Objectives & Goals

What this initiative needed to solve.

Create a scalable AI-document system that converts Rhode Island policy PDFs into reliable structured JSON with precise field coordinates, while minimizing manual workload and preventing unsafe model drift.

• Extract Every Input Reliably — Detect all form fields, including text entries, yes/no choices, and checkboxes, with deterministic field boundaries.
• Classify with Context — Attach each field to a meaningful section heading and program context so downstream LLMs can act with stronger semantic grounding.
• Prevent Hallucinations — Use multi-pass model consensus and rule-level post-checks before finalizing any annotation output.
• Enable 24/7 Processing — Deliver continuous, unattended PDF annotation throughput across all supported benefit documents.
• Reduce Operational Labor — Replace repetitive manual form parsing and QA cycles with an auditable, automated ingestion pipeline.

Key Features & Innovations

AI document intelligence designed for government-grade reliability.

PDF Layout + Text Extraction

We start with PDF parse and layout extraction to preserve reading order, page boundaries, and coordinate data needed for precise annotation.

Structured Output Contract

• Field geometry: Every input returns XY values with width and height metadata for deterministic downstream consumption.
• Type metadata: Each annotation includes input type and heading context, enabling programmatic interpretation.

Multi-Model LLM Consensus

• Independent analysis passes: Separate model views reduce single-source errors and make uncertain fields detectable.
• Validation layer: Cross-checking and rule filters catch impossible defaults, missing headings, and malformed coordinates before publish.

Checkbox Intelligence

Checkbox fields are enriched with inferred default state and confidence signals, which improves automation quality for decision trees and eligibility workflows.

Human-in-the-Loop Escalation

Low-confidence or conflicting annotations are flagged for review, while high-confidence fields flow automatically to the annotation store.

LLM Agent Compatibility

• Subsidy-ready artifacts: The JSON shape supports direct ingestion by AI assistants, extraction validators, and eligibility automation.
• Program scale: Built around major aid documents in Rhode Island so policy teams can process multiple programs from one pipeline.

Results & Impact

The pipeline reduced repetitive manual form-processing overhead and made policy document ingestion predictable at scale.

By forcing consensus-driven validation before output, teams gained higher confidence that extracted fields were trustworthy and consistent across programs.

The solution now powers eligibility and support AI agents with form-level context instead of loosely parsed plain text.

What our clients say

The annotation pipeline finally made our benefit PDF workflow dependable. We can now process policy documents for multiple programs with far less manual correction and much stronger confidence in every structured output.

Ito Health — Data Operations Lead

Technologies & Tools Used

LLM-powered policy document intelligence

Our approach combines PDF parsing, multi-pass large language model analysis, and schema-driven post-processing to convert policy forms into structured annotations with location metadata and high semantic confidence.

The stack is designed to prevent hallucinations through redundancy, validation, and confidence gating, while remaining fast enough for production scheduling across a full benefits document portfolio.

Technology stack

• pdf-parse — Core engine for extracting text blocks and layout-adjacent signals from PDF policy documents.
• Python — Orchestration layer for parsing workflows, annotation normalization, and output generation.
• OCR Service — Fallback text extraction for scanned or low-quality government PDFs with inconsistent text layers.
• Multi-LLM Strategy — Parallel model passes used for field detection, type classification, and annotation reconciliation.
• JSON Schema Validation — Strict contract enforcement to ensure each field includes heading, type, XY geometry, and checkbox state when applicable.
• PostgreSQL / JSON Store — Storage for processed annotation records and confidence metadata used by downstream AI agents.