What running Harvey LAB taught us about document fidelity

We ran a legal AI benchmark twice. Same tasks, same model, same grader. The scores came out identical. The documents did not.

In half of the marked-up contracts from the baseline run, every bold heading, every italic, every underline and every table was gone. Fee schedules. Underwriter allocations. Certificate rate grids. The grader passed all of them, because the grader reads text. A lawyer does not receive text. A lawyer receives a document.

Disclosure before anything else: we build Adeu, an open-source redlining engine for Word documents, and one of the two arms in this comparison is ours. Every number below is measured from the delivered files rather than reported by the grader, and we name the documents so you can open them yourself.

Why we looked at Harvey LAB

Harvey publishes LAB, the Legal Agent Benchmark: an open-source dataset of realistic legal tasks, each one a matter folder, an instruction, and a rubric of pass or fail criteria, plus a harness for running and scoring agents against them. You can read their announcement post for the background. It is one of the few serious public attempts to measure whether AI can do legal work rather than answer legal trivia, and that is exactly why we wanted to run it.

We had a question our own demos could not answer honestly: when a document-editing task is put in front of a capable AI agent, how much does the document layer actually matter?

So we set up a controlled A/B.

• Arm A, the baseline. Harvey LAB exactly as it ships. General-purpose agent tools: shell, read, write, edit, search, plus the repository's own document scripts.
• Arm B, Adeu. The identical harness, the identical model, the identical tasks, with Adeu replacing the document-editing layer. Not added alongside it. The question was what changes when the edit layer is purpose-built, not whether more tooling helps.

Same tasks. Same model. Same grader. One variable.

We expected to end up arguing about speed and cost. We found something we were not looking for.

Setup. A 215-task curated redlining corpus from LAB. Agent gemini-3.6-flash at high reasoning, judge gemini-3.6-flash, evaluated with --docx-track-changes all so tracked insertions, deletions and comments are visible to the grader in both arms. Each arm runs in its own sandboxed container with networking disabled. Rubric scoring is all-pass: a task scores 1.0 only if every criterion passes. The comparison below covers the 36 tasks where both arms produced completed runs, across 127 delivered .docx files.

What came back

On the tasks that required a marked-up contract, 32 such documents per arm:

"Stripped" here is not a subtle degradation. In the affected documents essentially all character and paragraph formatting is gone: bold, italics, underline, fonts, type sizes, indentation, justification, line spacing, and every table.

And every one of those documents passed its grading rubric. All 18 tasks with damaged deliverables scored a full pass on every criterion.

One contract, in detail

A stock purchase agreement. Counting formatting runs in the source, the baseline output, and the Adeu output:

SourceEnlarge

Baseline outputEnlarge

Adeu outputEnlarge

The cover page reads STOCK PURCHASE AGREEMENT in flat body text. Every "Section 1.1, Defined Terms" heading is indistinguishable from the paragraph beneath it.

The tables that disappeared

This is the part that matters commercially. The tables that vanished were not decorative:

• Transition services agreement: the Monthly Fee schedule and the Maximum Initial Term schedule. The commercial terms of the deal.
• Underwriting agreement: the Underwriter Allocations table. Who buys how many shares.
• Pooling and servicing agreement: the certificate class table, with initial balance, percentage and preliminary rating, plus the certificate rate table. The capital structure.
• Master supply agreement: the pricing exhibit and the purchase-order forms.

SourceEnlarge

Baseline outputEnlarge

Adeu outputEnlarge

In the transition services agreement the loss went further than formatting. The returned document held 27% of the source's text. Three-quarters of the agreement was gone, and it still passed every criterion.

The same pattern shows up in the master supply agreement, where Exhibit B is a pricing schedule:

SourceEnlarge

Baseline outputEnlarge

Adeu outputEnlarge

Every document we examined

Five measures across all six documents, counted directly from the XML of each delivered file. Same three columns as above, one group per document. The final group is the control, where the baseline agent avoided the lossy path and its output came back clean.

In five of the six documents the baseline column reads zero on every formatting measure, while the Adeu column returns to the source. Adeu's bold counts land slightly above the source because a tracked insertion inherits the formatting of the text it sits in, so a new clause inside a bold heading adds a bold run of its own.

Why this matters in practice

A redline is not a summary of edits. It is the artifact you send to the other side.

• It has to be sendable. A contract whose headings are not headings and whose fee schedule has vanished cannot go to opposing counsel, and cannot go to a client.
• The tracked changes were correct. That is what makes this dangerous. The agent's legal reasoning was sound, the edits were the right edits, and the document is still unusable. The failure is invisible in every text-based view, which is exactly how a reviewer would check it.
• Tables carry the money. Fee schedules, allocation tables, rate grids. A silent table deletion in a negotiation draft is a commercial error, not a cosmetic one.
• Benchmarks measure what they can read. The grader converts the document to text and checks whether the right things were said. Fidelity is not in the rubric, so a document that fails a lawyer's first glance can score 100%.

The gap here is not "the AI got the law wrong." It is that we are measuring legal AI on what it says, and not on what it hands you.

Why it happens

Both arms preserve the document's style definitions. The stylesheet, headers, footers, numbering and theme survive intact in close to 100% of outputs in both arms. That is precisely why this is easy to miss: unzip the file, see the styles, conclude it is fine.

The damage is one layer down. The style definitions survive, and every application of them in the document body is stripped. The accurate phrase is applied formatting stripped, not "styles stripped."

The baseline arm's damage traces to a single reference script in the LAB repository that produces a tracked-changes document by diffing two files. It reads each paragraph as a plain string, which discards run-level formatting at the moment of reading and never sees text inside tables at all. It then clears the document body and rebuilds it from the diff, emitting bare paragraphs containing bare runs. Tables are never rebuilt. The comment above the function notes that using the original as the base document preserves styles, headers and footers, which is true at the file level, and is why the loss is silent.

The correlation across the corpus is clean: 15 of 15 flattened runs invoked that script, and 0 of 9 runs that never invoked it were flattened. Twelve further runs called it, saw the result, and shipped output from a different path instead. So roughly half the agents that reached for the obvious tool got a broken document out of it.

An important clarification. This is a script in Harvey's open-source benchmark harness. It is not Harvey's commercial product, and we have not tested Harvey's commercial product. Nothing here should be read as a claim about it. The honest framing is that a capable agent, given general-purpose document tooling, produces documents that lose formatting, and nobody notices because nothing measures it.

Adeu does not have this failure mode because it applies edits in place on the existing document, inserting tracked-change markup around the existing runs rather than regenerating the body from text. Runs keep their properties because they are never recreated. The retention medians in the table above are worth reading against the source documents themselves, which score 0.93 and 0.98 on the same measures. Adeu matches them exactly, which is to say the only formatting change is the noise of the agent's own edits.

The secondary finding: newly written documents

Many LAB tasks also ask for a companion memo. Across those, 79% of baseline memos contain no heading structure at all and 47% have no styled paragraphs: flat walls of text. In the Adeu arm, no memo lacked headings, with a median of 22 headings and 190 styled paragraphs.

One caveat worth stating plainly: that difference comes from the Adeu arm's document-authoring instructions routing memo creation through a proper generator, not from the redlining engine. It is a separate result from the redlining finding and should not be folded into it.

What this does not show

The strongest thing we can do with a result that favours our own product is tell you where it is weak.

• The sample is favourable to the baseline. These 36 tasks were selected because the baseline arm passed all of them. The 50% figure describes those 36 completed pairs. It is not a random-sample estimate of the full corpus, and it should not be read as "half of all legal redlining tasks."
• One model, one run per task. Everything here is gemini-3.6-flash, with no variance measurement. Do not generalise it across models.
• The grader is the same model as the agent, grading its own arm's output. Arm against arm is reasonably protected, because self-preference applies equally to both. Absolute pass rates are not trustworthy and we are not quoting them as such.
• We have not compared quality or accuracy between the arms. The sample selection makes that comparison meaningless, and it is not the story here.
• Not all baseline outputs were damaged. On a management rollover agreement, the baseline agent avoided the lossy path entirely and its output was clean: 279 bold passages became 297, and all three tables survived. That is the honest picture. It is a coin flip, not a guarantee.

We are also holding back the speed and cost comparison. Those numbers exist, but the paired analysis has not been reviewed to the standard of the fidelity numbers above, and the older efficiency figures came from a superseded configuration. That is the subject of the next post in this series, once the comparison has been run and sanity-checked.

Document fidelity belongs in the rubric

The finding is not that one tool is better than another. It is that a whole dimension of correctness is currently going unmeasured in legal AI evaluation, and it happens to be the dimension a practising lawyer notices first.

If an evaluation converts every deliverable to text before grading it, then it is not measuring the deliverable. It is measuring a transcript of the deliverable. For legal work, where the document is the product, that gap matters.

This is why our own DOCX benchmark scores formatting fidelity and file integrity alongside task success, not just whether the requested edit was made. We would like to see fidelity become a scored axis in legal agent evaluation generally, including in LAB, and we are glad to share how we measure it.

The engine used in the second arm is MIT licensed and public at github.com/dealfluence/adeu. The benchmark is public at github.com/harveyai/harvey-labs. If you want to check any number in this post, the documents named above are the ones to open.

Citation

LAB is MIT licensed, and Harvey asks that it be cited in research that uses it. This post does, so for the record:

Harvey AI. Harvey LAB: The Legal Agent Benchmark, version 1.0, 2026. Repository at github.com/harveyai/harvey-labs, announced in Introducing Harvey's Legal Agent Benchmark.

Related articles