Document extraction needs a human review queue

Document extraction is not complete when a model returns JSON. A production workflow must decide which document type arrived, whether required pages are present, which fields were found, where each value came from, whether the values agree with business rules, and what happens when confidence is insufficient.

Human review should not mean rereading every document. The system should complete routine work automatically and present only specific exceptions with the evidence needed to decide. The review queue is therefore part of the product architecture, not a temporary safety net to remove after the demo.

Validate fields independently from model confidence

A model’s confidence is not the same as business validity. An invoice total may be extracted clearly but disagree with its line items. An identity number may match the visible text but fail its checksum. A contract date may be present but fall outside the permitted period. Deterministic validation should run after extraction and produce explicit reasons for review.

Each field should retain provenance: document identifier, page, region or supporting text, extraction version, and any transformation applied. An operator needs to see the proposed value next to its source rather than search a long document from the beginning.

Different fields deserve different thresholds. A missing optional note is not equivalent to an uncertain bank account or legal entity. Review policy should reflect the consequence of accepting a wrong value, not one global confidence number.

Design the queue around decisions

A useful queue groups documents by the action required: confirm one field, resolve conflicting sources, select a document type, request a missing page, or reject the submission. It shows why the item was routed, how long it has waited, who can decide, and what downstream process is blocked.

Corrections should become structured feedback. Record the original output, corrected value, reason, source evidence, and model or pipeline version. These examples improve evaluation and can reveal that a template, parser, or business rule needs attention; they should not automatically train a model without review.

In one anonymised document workflow, reviewers originally reopened the full document even when only one value was uncertain. The useful change was not another model prompt. It was a queue item that named the disputed field, explained the routing reason, and placed the proposed value beside its source. The operator could make the business decision without reconstructing the entire extraction process.

Measure automation without hiding risk

Useful metrics include field accuracy by document type, review rate, false acceptance, false rejection, time to decision, and the causes of exceptions. A high automation rate is not a success if the remaining errors are concentrated in the most consequential fields.

The target is controlled throughput: routine documents move quickly, unusual documents receive focused human judgement, and every accepted value can be traced to evidence. That is a document system a business can operate, rather than a JSON demonstration with unresolved responsibility.