Automated data extraction: the hard part is verification

The short version

• Automated data extraction is a schema problem before it is a model problem: you cannot extract a field nobody has defined.
• Fixed layouts are cheap to automate with parsing and OCR. Varied layouts are where a language model earns its cost.
• The common failure is not a blank field. It is a correctly formatted value taken from the wrong place.
• Per-page extraction pricing is rarely the dominant cost. The engineering time around it is.
• If you cannot say which extractions to trust, you have not automated extraction, you have moved the reading.

What does it mean to automate data extraction?

It means turning documents, pages or messages into named fields that match a schema you defined first, without a person reading the source every time. The extraction step is the part everyone pictures. The schema and the verification are the parts that decide whether it works.

Three things make up any extraction system. The input, which is whatever arrives and however much it varies. The schema, which is the exact list of fields, their types and the rules they obey. The verification, which is how you find out an extraction is wrong before something downstream acts on it. Most projects build the middle one and assume the other two.

'We want to pull everything off the invoice' is an awkward place to start, because everything is not a field. Supplier name, VAT number, invoice date, net, VAT, gross, and line items with quantity and unit price: that is a field list. Write it down before choosing any tool, because the list decides which tools are even candidates.

Which extraction method fits your documents?

How much the input varies decides it, not how advanced the tool is. If every file has the same layout, parse it directly and keep the model out. If the same information shows up in a hundred different layouts, that variation is exactly what a language model is for.

Digital PDFs with a consistent structure do not need OCR at all, because the text is already in the file. Scanned pages with fixed boxes do well with OCR plus a template. Documents that vary by sender need a model that reads meaning rather than position. Web pages sit in between: CSS selectors are cheap and break often, and a model costs more per page but survives a redesign.

Most working systems use more than one. A cheap path handles the eighty per cent that look alike, and anything failing the cheap path falls through to the expensive one. That layering is where the economics live, and it is why a single tool choice is the wrong first question.

Which approach suits which input?

Should you buy an extraction tool or build a pipeline?

Buy first, if your document type is common. Invoices, receipts, identity documents, CVs and shipping paperwork are all served by vendor APIs that have seen far more of them than you ever will, and you will not beat them on price per page for a standard field set.

Build when the fields are specific to your business, when the documents are in a format nobody else uses, when the data cannot leave your environment, or when extraction is only the first step of a process that has to run anyway. The last of those is the usual reason. Extraction on its own rarely delivers anything, it feeds something else.

The practical middle is common: a vendor API for the reading, your own code for the schema, the checks, the corrections and the hand-off. That keeps the part you cannot buy, which is the business logic describing what a correct record looks like.

What does automated data extraction cost?

Less per page than most people expect, and more in engineering than most people budget. AWS publishes basic text detection at $0.0015 per page for the first million pages a month in its Oregon region, and prices its invoice and receipt API separately. At that rate, ten thousand pages of plain text detection costs fifteen dollars.

Set that against the people. The US Bureau of Labor Statistics publishes a median annual wage for software developers, and notes it is a wage figure only, so it understates what an employer actually pays once everything else is loaded on. Against any realistic version of that number, the per-page bill is not what decides whether the project is worth doing. The build, the corrections and the maintenance are.

The third cost is the one nobody prices: being wrong. A wrong total on a paid invoice, a wrong dosage in a record, a wrong postcode on a shipment. Work out what a single bad field costs you before deciding how much verification to pay for, because that number sets the budget for everything else.

How do you know the extracted data is right?

You check it against something else, and you decide in advance what happens when the two disagree. A confidence score from the model is not that check, because a model can be confident and wrong in exactly the same format as when it is confident and right.

The most careful published design here comes from medicine. A prospective study within six AHRQ systematic reviews extracted 9,341 data elements from 63 studies twice in parallel, once by an AI-assisted team using Claude with a human verifying every single extraction, and once by a human-only team, with both scored against data finalised by blinded independent adjudicators. Note the shape of that design: human verification sits inside the AI arm, not bolted on afterwards as a fallback.

The same study recorded time on task for both arms, and the authors flagged that consistent time tracking was a limitation of the work. That is worth remembering when somebody quotes you a time saving, including when the somebody is your own pilot. Measure your own before and after, on your own documents, or treat the number as decoration.

The checks that survive production are boring ones. Make the arithmetic prove itself, so line items have to sum to the stated total. Cross-reference against a system you already trust, so a purchase order number has to exist. Extract a sample twice by different methods and compare. Route anything that fails to a person, with the source page beside the field, so a correction takes seconds rather than minutes.

What to settle before you write any code

• •The exact field list, with types, units and formats.
• •What an absent field means, and whether null is allowed to reach the next system.
• •Which fields are safe to be wrong, and which must be correct every time.
• •Where a rejected extraction goes, and whose job it is to look at it.
• •How you will measure accuracy in six months without re-reading everything by hand.

Is automating extraction worth it for a small team?

Usually yes for one high-volume document type with little variation, and usually no for a long tail of exceptions that each arrive twice a year. The second case is where these projects die, because the exceptions take most of the effort and save almost none of the time.

Text analysis is already the most common production use of AI in European firms. Eurostat's 2025 survey of EU enterprises with ten or more employees found text mining was the most used AI technology, at 11.75% of enterprises. In the US, a survey published by the Chamber of Commerce's technology engagement centre put self-identified generative AI use among small businesses at 40% in 2024, up from 23% in 2023.

Adoption is not evidence that it will work for you, and neither figure says anything about whether those deployments are accurate. Use them for one thing only: this is ordinary now, so the question is not whether it can be done but which single document type to start with. Pick the one that arrives most often and varies least, get it correct end to end, then take the next one.