Contract Review AI: Matches Junior Lawyers in Tests, but the Evidence Is Thin

The use case

In a January 2024 preprint, the best-scoring of six language models on issue determination reviewed a procurement contract in an average of 4.7 minutes at an average cost of $0.25, against 56.17 minutes and $74.26 for a junior lawyer (measured on ten contracts, not replicated)¹. That comparison is the core of the use case. Software reads an agreement, flags clauses that depart from a checklist of standard positions (a "playbook"), extracts fields such as dates, parties and liability caps, and in some products proposes edits (redlining).

Scope note. AI here means machine learning, including large language models (LLMs, systems trained on very large volumes of text to analyse and generate language). Older approaches that do similar work are not credited as AI: keyword search, fixed clause libraries and templates, and human services such as junior lawyers and legal process outsourcers (LPOs, firms that carry out routine legal review at lower cost). Contract drafting, legal research and e-discovery are out of scope. One legal-research study is used only as a warning about tool reliability⁴.

Where it is running

Three limits apply to this picture. The A&O Shearman announcements describe a general assistant and a loan review agent; neither names contract review as a use case⁶⁷. The survey lists document review (74%) and contract drafting (51%) among use cases but does not name contract review, and the base for those percentages was not established⁸. The Vals Legal AI Report names four commercial tools (CoCounsel, Vincent AI, Harvey Assistant and Oliver) that were tested on document tasks including data extraction and redlining in February 2025². No deployment in continental Europe or Asia was established, so the evidence covers the United States and the United Kingdom only.

What the evidence shows

Benchmark against a lawyer baseline. The Vals Legal AI Report (27 February 2025) tested vendor tools on seven tasks using over 500 samples, mostly from large US law firms, against an "average" lawyer baseline sourced through an alternative legal service provider. An automated AI judge did the scoring, and vendors opted in task by task (measured in a benchmark, United States only)². On data extraction, 30 questions across 29 documents, the lawyer baseline scored 71.1%, Harvey Assistant 75.1%, CoCounsel 73.2%, Vincent AI 69.2% and Oliver 64.0%². With a gap of four points on a small set, parity is the safer reading than superiority. On the hardest question, a 40-field extraction, the lawyer baseline scored 39.2% and the tools ranged from 11.7% to 61.7%². On redlining, 20 samples, the lawyer baseline scored 79.7%, Vincent AI 53.6% and Harvey Assistant between 59.4% and 65.0% (the report gives both figures)². The tools were between six and 80 times faster than the lawyers (measured in the same benchmark)².

Head-to-head contract test. In the January 2024 preprint, ten procurement contracts were reviewed against a playbook, and a consensus of senior lawyers set the benchmark answers¹. For deciding whether a contract met or departed from the playbook, the F-score (a combined measure of precision and recall, where 1.0 is perfect) was 0.874 for LPOs, 0.871 for GPT4-1106 and 0.860 for junior lawyers (measured without independent replication). For locating the clause behind each judgment, LPOs scored 0.770, the best model (GPT4-32k) 0.740 and junior lawyers 0.667¹. Average cost per contract was $0.02 to $1.24 for the models, $36.85 for LPOs and $74.26 for junior lawyers, and tuning prompts took an average of 16 hours¹. The authors work for Onit, a legal technology company, and the paper is an arXiv preprint with no stated peer review¹.

Why the cost gap overstates the saving. The per-contract figures count model fees for 2023-era models. They exclude the 16 hours of set-up, any subscription and the lawyer time needed to check outputs. The authors themselves say the optimistic outlook should be tempered by the need for ongoing supervision of model outputs¹.

Where the field started. In March 2021 the CUAD dataset was published with over 13,000 annotations made with dozens of legal experts, and its abstract found that Transformer models had "nascent performance"³. The abstract names the cost of expert annotation as the barrier. Current products rely on LLMs that need far less labelled data, which is why the 2024 and 2025 tests look so different from 2021 (analysis, not a measured finding).

The head-to-head evidence comes from two tests, and neither is free of vendor ties. Both tests are small, and neither followed any contract after review to see whether flagged or missed clauses mattered.

What it takes: cost, integration, governance

Cost. No vendor price list was opened, so subscription prices are not established. A&O Shearman states that its agents are available by subscription or usage-based fees, with the firm sharing in software revenue⁷. The only public per-contract figures are the model fees in the preprint¹.

Integration. A tool must be set up against a playbook of the organisation's standard positions, and the preprint reports an average of 16 hours of set-up, testing and prompt tuning¹. The Vals report notes that its evaluators used Harvey's general assistant rather than its redlining feature, so product configuration affects results².

Professional duties. The American Bar Association's Formal Opinion 512 (29 July 2024) lists duties that apply to generative AI use: competence, confidentiality, communication, supervision, advancing only meritorious claims, candour toward the tribunal and reasonable fees⁵. It says lawyers "may not abdicate their responsibilities by relying solely on a GAI tool to perform tasks that call for the exercise of professional judgment", and that informed client consent is required before inputting client information into self-learning tools⁵. In England and Wales, the Law Society published a guide on managing AI risks, reported on 25 March 2026; its content was not read beyond that report⁹.

EU rules. The EU AI Act (Regulation (EU) 2024/1689) treats some narrow procedural tasks, "such as an AI system that transforms unstructured data into structured data", as a condition for not regarding a listed system as high-risk (recital 53)¹⁰. Whether any contract-review tool is classed as high-risk was not established, because the Annex III wording on administration of justice was not read in full. Regulation (EU) 2026/1744 of 8 July 2026, the Digital Omnibus on AI, amends the Act¹¹. A non-official copy of Article 113 shows high-risk obligations for Annex III systems applying from 2 December 2027¹².

Where it falls short

Challenge question. The usual claim is that AI reviews contracts at lawyer-level accuracy for a fraction of the cost. Tested against its design, that claim is weaker than it sounds.

• Sample size. The preprint used ten contracts and did not report how many reviewers were in each comparison group¹. The Vals extraction set had 30 questions and the redlining set 20 samples².
• Who ran the tests. The preprint authors work for a legal technology company¹. Vals states it has "a customer relationship with one or more of the participants"².
• Selection. Vendors opted in task by task. Lexis+ AI withdrew from the sections studied, Harvey withdrew from EDGAR research and vLex from chronology generation, so published scores reflect the tasks vendors chose².
• Grading. Vals scored with an automated AI judge and states that this technology "is not sufficient to replace an expert human grader"².
• Baselines. The lawyer baseline was "average" lawyers working over two weeks². The preprint authors say the study does not cover a broad enough range of junior lawyer work for a one-to-one comparison¹.
• Uneven results. Redlining trailed the lawyer baseline by 15 to 26 points in the Vals report, and the hardest extraction question produced tool scores from 11.7% to 61.7%².
• Reliability in adjacent tools. A Stanford study of AI legal research tools, published in 2025, found they hallucinated between 17% and 33% of the time on 2024 product versions (abstract)⁴. That study concerns legal research, not contract review, and does not transfer directly. It does show that vendor claims of reliability needed independent testing.

Searches for failures, lawsuits, withdrawals and regulator action specific to AI contract review found no documented incident or enforcement action in the sources opened. That absence reflects limited public reporting and is not evidence of safety. No evidence was found on whether gains persist over time or reach smaller organisations.

Phase: In early use — why

The phase hypothesis of "In wide use" is not kept. All named deployments use language models, so machine learning is confirmed. The scale of use for contract review specifically could not be established. The only adoption share found covers generative AI in general, at 26% of respondents, and does not name contract review⁸. The largest named deployment is a general assistant offered to about 3,500 lawyers (operator-reported)⁶. The lower of the plausible phases is therefore chosen. Independent adoption data for contract review would support a move to "In wide use".

What to watch next

• Later editions of the Vals Legal AI Report, which describes the February 2025 study as the first of a regular series with a planned UK expansion². The trigger is a newer edition with contract tasks and published vendor withdrawals.
• 2 December 2027, the date from which high-risk obligations for Annex III systems apply under the amended AI Act, per a non-official copy of Article 113¹¹¹².
• A peer-reviewed version or independent replication of the January 2024 preprint, with reviewer numbers reported¹.

Questions to ask before adopting

• Which contract types and clauses was the tool tested on, and how did it score against your own playbook?
• Who ran the test, and do they have a commercial tie to the vendor?
• What is the all-in cost per contract once set-up, subscription and lawyer checking are included?
• Who signs off each flagged and unflagged clause, and how would a missed clause be detected?
• Where is client data processed, and does your professional-conduct rule require client consent?