What running Harvey LAB taught us about document fidelity

Two AI agents worked the same 36 tasks: real contracts, real redlining instructions, same model, same harness, same grader. One returned documents a lawyer could send, bold headings, tables and fee schedules all in place, tracked changes applied. The other returned the same words with half of its marked-up contracts stripped bare: no formatting and, in 13 of them, no tables at all.

The grader could not tell them apart. The scores came out identical, because the grading reads text, and the text was fine. A lawyer does not receive text. A lawyer receives a document.

Disclosure before anything else: we build Adeu, an open-source redlining engine for Word documents, and one of the two configurations in this comparison is ours. Every number below is measured from the delivered files rather than reported by the grader, and we name the documents so you can open them yourself.

Why we looked at Harvey LAB

Harvey publishes Harvey LAB, its Legal Agent Benchmark: an open-source collection of realistic legal tasks, each one a matter folder, an instruction, and a set of pass or fail criteria, plus a harness for running and scoring agents against them. You can read their announcement post for the background. It is one of the few serious public attempts to measure whether AI can do legal work rather than answer legal trivia, and that is exactly why we wanted to run it.

Two pieces of benchmark vocabulary, because the rest of this post depends on them:

• Rubric. The list of pass or fail criteria attached to a task. An AI judge reads the agent's output and checks it against each criterion. In Harvey LAB the scoring is all-pass, so a task scores 1.0 only if every criterion passes.
• Arm. One of the two setups being compared in a controlled test. Everything is held identical between them except the single thing under test.

Harvey LAB is large and covers many kinds of legal work. It holds roughly 1,760 tasks today, and it is still growing. We did not run all of it. We selected the redlining tasks: the ones where the agent has to hand back a marked-up contract, and the ones where it has to read a document that already carries tracked changes. That is where our question lives, and a benchmark that grades text cannot tell you much about a file format it never inspects.

We had a question our own demos could not answer honestly: when a document-editing task is put in front of a capable AI agent, how much does the document layer actually matter?

So we set up a controlled A/B on those tasks.

• Arm A, general-purpose tooling. Harvey LAB's redlining tasks exactly as the benchmark ships them. General-purpose agent tools: shell, read, write, edit, search, plus the reference document scripts included in the repository. This is the arm we call the baseline throughout this post.
• Arm B, Adeu. The identical harness, the identical model, the identical tasks, with Adeu replacing the document-editing layer. Not added alongside it. The question was what changes when the edit layer is purpose-built, not whether more tooling helps.

Same tasks. Same model. Same grader. One variable.

What this comparison is, and what it is not. The baseline arm uses the reference document scripts published in Harvey's open-source benchmark repository. It is not Harvey's commercial product, and we have not tested Harvey's commercial product. Nothing here tells you how Harvey redlines a contract or reads a .docx for a paying customer, and none of it should be read that way. What this comparison does show is narrower and, we think, more useful: what a capable agent hands back when its document layer is general-purpose, and how much a tool built specifically for the job changes the result.

We expected to end up arguing about speed and cost. We found something we were not looking for.

Setup. A 215-task curated redlining corpus drawn from Harvey LAB. Agent gemini-3.6-flash at high reasoning, judge gemini-3.6-flash, evaluated with --docx-track-changes all so tracked insertions, deletions and comments are visible to the grader in both arms. Each arm runs in its own sandboxed container with networking disabled. The comparison below covers the 36 tasks where both arms produced completed runs, across 127 delivered .docx files.

What came back

On the tasks that required a marked-up contract, 32 such documents per arm:

"Stripped" here is not a subtle degradation. In the affected documents essentially all character and paragraph formatting is gone: bold, italics, underline, fonts, type sizes, indentation, justification, line spacing, and every table.

And every one of those documents passed its grading rubric. All 18 tasks with damaged deliverables scored a full pass on every criterion.

One contract, in detail

A stock purchase agreement. Counting formatting runs in the source, the baseline output, and the Adeu output:

SourceEnlarge

Baseline outputEnlarge

Adeu outputEnlarge

The cover page reads STOCK PURCHASE AGREEMENT in flat body text. Every "Section 1.1, Defined Terms" heading is indistinguishable from the paragraph beneath it.

The tables that disappeared

This is the part that matters commercially. The tables that vanished were not decorative:

• Transition services agreement: the Monthly Fee schedule and the Maximum Initial Term schedule. The commercial terms of the deal.
• Underwriting agreement: the Underwriter Allocations table. Who buys how many shares.
• Pooling and servicing agreement: the certificate class table, with initial balance, percentage and preliminary rating, plus the certificate rate table. The capital structure.
• Master supply agreement: the pricing exhibit and the purchase-order forms.

SourceEnlarge

Baseline outputEnlarge

Adeu outputEnlarge

In the transition services agreement the loss went further than formatting. The returned document held 27% of the source's text. Three-quarters of the agreement was gone, and it still passed every criterion.

The same pattern shows up in the master supply agreement, where Exhibit B is a pricing schedule:

SourceEnlarge

Baseline outputEnlarge

Adeu outputEnlarge

Every document we examined

Five measures across all six documents, counted directly from the XML of each delivered file. Same three columns as above, one group per document. The final group is the control, where the baseline agent avoided the lossy path and its output came back clean.

In five of the six documents the baseline column reads zero on every formatting measure, while the Adeu column returns to the source. Adeu's bold counts land slightly above the source because a tracked insertion inherits the formatting of the text it sits in, so a new clause inside a bold heading adds a bold run of its own.

Why this matters in practice

A redline is not a summary of edits. It is the artifact you send to the other side.

• It has to be sendable. A contract whose headings are not headings and whose fee schedule has vanished cannot go to opposing counsel, and cannot go to a client.
• The tracked changes were correct. That is what makes this dangerous. The agent's legal reasoning was sound, the edits were the right edits, and the document is still unusable. The failure is invisible in every text-based view, which is exactly how a reviewer would check it.
• Tables carry the money. Fee schedules, allocation tables, rate grids. A silent table deletion in a negotiation draft is a commercial error, not a cosmetic one.
• Benchmarks measure what they can read. The grader converts the document to text and checks whether the right things were said. Fidelity is not in the rubric, so a document that fails a lawyer's first glance can score 100%.

The gap here is not "the AI got the law wrong." It is that we are measuring legal AI on what it says, and not on what it hands you.

Why it happens

Both arms preserve the document's style definitions. The stylesheet, headers, footers, numbering and theme survive intact in close to 100% of outputs in both arms. That is precisely why this is easy to miss: unzip the file, see the styles, conclude it is fine.

The damage is one layer down. The style definitions survive, and every application of them in the document body is stripped. The accurate phrase is applied formatting stripped, not "styles stripped."

The baseline arm's damage traces to a single reference script in the Harvey LAB repository that produces a tracked-changes document by diffing two files. It reads each paragraph as a plain string, which discards run-level formatting at the moment of reading and never sees text inside tables at all. It then clears the document body and rebuilds it from the diff, emitting bare paragraphs containing bare runs. Tables are never rebuilt. The comment above the function notes that using the original as the base document preserves styles, headers and footers, which is true at the file level, and is why the loss is silent.

The correlation across the corpus is clean: 15 of 15 flattened runs invoked that script, and 0 of 9 runs that never invoked it were flattened. Twelve further runs called it, saw the result, and shipped output from a different path instead. So roughly half the agents that reached for the obvious tool got a broken document out of it.

An important clarification, worth repeating here. This is a reference script in Harvey's open-source benchmark repository. It is not Harvey's commercial product, we have not tested Harvey's commercial product, and we are not claiming this is how Harvey redlines documents or parses .docx files for its customers. The finding is about the script, not the company. The honest framing is that a capable agent, handed general-purpose document tooling, produces documents that lose formatting, and nobody notices because nothing measures it.

Adeu does not have this failure mode because it applies edits in place on the existing document, inserting tracked-change markup around the existing runs rather than regenerating the body from text. Runs keep their properties because they are never recreated. The retention medians in the table above are worth reading against the source documents themselves, which score 0.92 and 0.98 on the same measures. Adeu lands at 0.93 and 0.98, which is to say the only formatting difference is the noise of the agent's own edits.

The secondary finding: newly written documents

Many of these tasks also ask for a companion memo. Across those, 79% of baseline memos contain no heading structure at all and 47% have no styled paragraphs: flat walls of text. In the Adeu arm, no memo lacked headings, with a median of 22 headings and 190 styled paragraphs.

One caveat worth stating plainly: that difference comes from the Adeu arm's document-authoring instructions routing memo creation through a proper generator, not from the redlining engine. It is a separate result from the redlining finding and should not be folded into it.

What this does not show

The strongest thing we can do with a result that favours our own product is tell you where it is weak.

• The sample is favourable to the baseline. These 36 tasks were selected because the baseline arm passed all of them. The 50% figure describes those 36 completed pairs. It is not a random-sample estimate of the full corpus, and it should not be read as "half of all legal redlining tasks."
• One model, one run per task. Everything here is gemini-3.6-flash, with no variance measurement. Do not generalise it across models.
• The grader is the same model as the agent, grading its own arm's output. Arm against arm is reasonably protected, because self-preference applies equally to both. Absolute pass rates are not trustworthy and we are not quoting them as such.
• We have not compared quality or accuracy between the arms. The sample selection makes that comparison meaningless, and it is not the story here.
• Not all baseline outputs were damaged. On a management rollover agreement, the baseline agent avoided the lossy path entirely and its output was clean: 279 bold passages became 297, and all three tables survived. That is the honest picture. It is a coin flip, not a guarantee.

Speed and cost are in Part 2. This post is about what the two arms hand back. That one is about what each of them costs to get there: runtime, token consumption and cost per delivered document, measured on the same paired runs and held to the same standard as the fidelity numbers above.

Formatting matters, and Adeu keeps it

Two things are true in the numbers above.

The first is that formatting is not cosmetic. It is the difference between a draft you can send to opposing counsel and a file you have to rebuild by hand. General-purpose document tooling flattened half the contracts it touched and deleted the tables that carried the money, and the score never moved.

The second is that a tool built for the job does not have this problem. Adeu returned 32 of 32 contracts intact, matching the source documents on every formatting measure we counted, with the tracked changes applied where they belonged. That is not a marginal gain over the general-purpose path. It is the difference between a deliverable and a transcript.

If an evaluation converts every deliverable to text before grading it, then it is not measuring the deliverable. For legal work, where the document is the product, that gap matters. This is why our own DOCX benchmark scores formatting fidelity and file integrity alongside task success, not just whether the requested edit was made. We would like to see fidelity become a scored axis in legal agent evaluation generally, including in Harvey LAB, and we are glad to share how we measure it.

The engine used in the Adeu arm is MIT licensed and public at github.com/dealfluence/adeu. The benchmark is public at github.com/harveyai/harvey-labs. If you want to check any number in this post, the documents named above are the ones to open.

Citation

Harvey LAB is MIT licensed, and Harvey asks that it be cited in research that uses it. This post does, so for the record:

Harvey AI. Harvey LAB: The Legal Agent Benchmark, version 1.0, 2026. Repository at github.com/harveyai/harvey-labs, announced in Introducing Harvey's Legal Agent Benchmark.

Related articles