AI Document Workflow Reliability Benchmark: Open Research Protocol

Published benchmark protocol v1.0 — 2026-08-09. No model has been selected and no result has been measured. The proposed dataset will be synthetic. Model IDs, prompts, code, raw outputs, prices, and test dates must be frozen before results are written.

An AI document workflow can look excellent in a demo and still create more work than it removes. The model may extract most fields correctly while confidently corrupting a small number of consequential values. It may succeed on clean PDFs and fail on rotated photos. It may produce valid JSON while assigning a phone number to the wrong record. It may appear inexpensive per request while creating an expensive review queue.

This benchmark evaluates the complete document-to-record workflow: input handling, extraction, validation, abstention, human escalation, correction, and recovery. The initial task is intentionally narrow enough to reproduce with synthetic data.

Key Findings

The current findings are methodological and will remain separate from future measured results:

1. Aggregate accuracy is not a deployment decision. A workflow needs field-level correctness, consequential-error rates, abstention behavior, latency, cost, and human-review burden.
2. The validator is part of the system. Schema checks, date/amount constraints, identifier patterns, duplicate detection, and cross-field rules can catch failures that a model-level score hides.
3. Safe failure needs a measurable definition. “Human in the loop” is too vague. The benchmark must report which documents are escalated, how often bad records pass automatically, and how many correct records are needlessly reviewed.
4. Synthetic data protects clients but narrows generalization. The first release can be open and privacy-safe, but it cannot prove performance on every real document distribution.

No model ranking, accuracy percentage, price, or time saving will appear until the frozen harness has produced raw output.

Methodology

Task definition

The proposed workflow converts a mixed-format small-business intake document into a structured record with these fields:

• Document type
• Organization/person name
• Email address
• Phone number
• Service or request category
• Request date
• Currency amount when present
• Free-text summary
• Source document identifier
• Extraction confidence or abstention state

The exact schema will be published as JSON Schema. Every output must either validate against it or enter an explicit failure state. The system may not silently drop an input.

Proposed dataset

Create 300 synthetic documents with deterministic ground-truth JSON records. No client or real-person data will be used.

Stratum	Proposed count	Purpose
Clean digitally generated PDF	60	Establish baseline extraction behavior
Scanned PDF with moderate noise	60	Test OCR and layout degradation
Mobile photo with rotation/perspective	60	Test common field-capture conditions
Email-style plain text/HTML	60	Test less structured but machine-readable input
Adversarial/ambiguous cases	60	Test missing fields, conflicting values, prompt-like text, and malformed layouts

The generator will vary names, phone formats, dates, currency formatting, field order, optional fields, duplicate documents, and plausible distractor values. Random generation will use a published seed so the corpus can be recreated.

The adversarial stratum is not a cybersecurity penetration test. It is a reliability set containing ambiguity and instruction-like document text that should not override the extraction task.

Systems under test

The initial study may compare:

• A rules-only baseline
• One or more current multimodal language-model APIs
• A hybrid workflow using OCR/rules before a language model
• The same model with and without deterministic validation/retry logic

Final systems will be selected immediately before collection because models and pricing change. The article will record provider, exact model identifier, API version, region if relevant, request parameters, prompt/template hash, structured-output mode, retry policy, and public price source on the test date.

No provider will be described as the “best AI model.” The result applies to this task, corpus, configuration, and date.

Run controls

1. Freeze the dataset, ground truth, schema, scorer, and primary metrics before running models.
2. Use a clean API session and the same input representation for comparable systems.
3. Set deterministic parameters where the API supports them and record unsupported controls.
4. Run each document once for the primary cost/latency analysis.
5. Run a pre-registered stability subset multiple times to measure output variation.
6. Preserve raw requests/responses after removing credentials and provider-restricted data.
7. Log retries, timeouts, schema failures, refusals, and provider errors.
8. Price each run from measured token/image usage and the provider's dated public price—not from memory.
9. Manually review scorer disagreements before final aggregation without changing ground truth after seeing a model label.

Primary metrics

Field exact-match accuracy

For normalized fields such as phone, email, date, amount, and category:

fieldAccuracy = correct extracted fields / scorable ground-truth fields

Normalization rules are published in advance. For example, phone formatting punctuation may be ignored while digits and extension must match.

Record perfect-match rate

recordPerfectMatch = records with every required field correct / all records

This deliberately produces a stricter view than field-level accuracy.

Undetected consequential error rate

Define consequential fields before testing—for example identity, contact destination, date, and currency amount.

undetectedConsequentialError = bad consequential records accepted automatically
                               / records accepted automatically

This is the primary safety metric for an auto-entry decision. A workflow that escalates every record could have a low undetected-error rate while providing no automation value, so it must be read with review burden.

Review burden

reviewRate = records sent to human review / all records
falseReviewRate = fully correct records sent to review / fully correct records

The final artifact will also time a defined review procedure on a sample. It will not invent universal wage savings.

Abstention quality

Treat an explicit “cannot determine” or validation failure as abstention. Report:

• Recall of bad records into review
• Precision of the review queue
• Coverage: percentage accepted automatically
• Risk-coverage curve as the confidence/review threshold changes

Latency and cost

Report median, 95th percentile, minimum, and maximum end-to-end latency. Report measured API cost per document and per 1,000 documents for this corpus and date. Keep provider pricing inputs visible so the calculator can be updated.

Acceptance scenarios

Instead of one arbitrary pass/fail line, publish three user-editable scenarios:

Scenario	Consequential error tolerance	Review capacity	Intended use
Draft assistance	User reviews every record	High	Summarization/data-entry aid
Guarded automation	Only flagged records reviewed	Medium	Low-risk operational intake
High-consequence workflow	Near-zero undetected error required	Variable	Not approved by this benchmark alone

The benchmark will not certify a high-consequence deployment. It shows the tradeoff curve and the evidence a decision-maker would still need.

Results table (blocked pending execution)

System	Field accuracy	Perfect records	Undetected consequential error	Review rate	Median latency	Cost/1,000
Rules baseline	Not measured	Not measured	Not measured	Not measured	Not measured	Not measured
AI workflow A	Not measured	Not measured	Not measured	Not measured	Not measured	Not measured
Hybrid workflow	Not measured	Not measured	Not measured	Not measured	Not measured	Not measured

Every aggregate will be accompanied by a breakdown for each input stratum. A strong clean-PDF result must not conceal failure on mobile photos or ambiguous documents.

Operational decision framework

After results exist, a business can use four gates:

1. Validity: Does the workflow produce the required schema and preserve every input?
2. Reliability: At the chosen automation coverage, is the undetected consequential-error rate acceptable for this task?
3. Operability: Can the business handle the review queue, outages, retries, corrections, and audit records?
4. Economics: Using its own labor inputs, does the measured review burden plus API/tooling cost improve the current process?

Passing the first three gates does not prove the fourth. The companion calculator will use measured review minutes and user-entered labor values rather than claiming a universal ROI.

Limitations

• Synthetic documents cannot represent every real layout, handwriting style, language, domain vocabulary, fraud pattern, or image condition.
• A 300-document corpus may be too small to estimate rare failure rates precisely.
• Model APIs, hidden system behavior, prices, and availability can change after the test date.
• Repeated API calls may not be deterministic even when temperature or seed controls are available.
• The benchmark measures one extraction workflow, not general reasoning ability.
• The defined “consequential” fields reflect the proposed task and must change for other business processes.
• A validator can catch malformed or implausible values but cannot guarantee that a plausible value belongs to the correct real-world entity.
• Privacy, retention, contractual, legal, and sector-specific requirements are outside the benchmark and need separate review.
• The study cannot authorize fully automatic use in financial, legal, medical, employment, safety, or other high-consequence decisions.

What Mendola.Tech adds

Mendola.Tech's contribution will be an open, end-to-end reliability harness built around operational decisions rather than a model-demo score. The repository will include the synthetic corpus generator, ground-truth records, JSON Schema, prompts/configuration hashes where permitted, scoring code, raw-output format, validation rules, review-threshold calculator, and reproducible reporting command.

The unique perspective comes from treating AI as one component in an automation system. API integration, schema validation, retries, human escalation, auditability, and failure recovery are measured alongside the model response because those are the parts that determine whether a workflow can actually be operated.

Sources

• NIST AI Risk Management Framework — National Institute of Standards and Technology; accessed 2026-08-09. Supports lifecycle risk management and incorporating trustworthiness into AI design, use, and evaluation.
• NIST AI Resource Center — NIST; accessed 2026-08-09. Supports testing, evaluation, verification, and validation as operational AI practices.
• NIST AI RMF Core — NIST; accessed 2026-08-09. Supports repeatable/documented evaluation, reliability validation, limitation documentation, safe failure, monitoring, and contextual interpretation.
• Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile — NIST AI 600-1; accessed 2026-08-09. Supports applying risk-management actions to generative-AI systems based on context, requirements, and risk tolerance.
• Effectiveness of the AI RMF — NIST AI Resource Center; accessed 2026-08-09. Supports evaluating whether processes, indicators, measurements, and expected outcomes actually improve AI risk management.