Two agents worked the same 215 legal tasks: real matter folders, real redlining instructions, the same model, the same harness, the same grader and the same scoring. The only difference between them was the tools each one used to open, read and edit Word documents. One finished the work 16% cheaper and 18% faster.
The reason is not that it read less. It is that it wrote less. To look inside a .docx file, the baseline agent typed roughly 59,000 characters of Python per task. The other one issued roughly 2,200 characters of tool arguments. A page number instead of a program.
Disclosure before anything else: we build Adeu, an open-source redlining engine for Word documents, and one of the two configurations here is ours. Every number below comes from 430 completed runs, all of which we report, including the 55 tasks where the baseline was cheaper and faster. The per-pair numbers are published as JSON so you can recompute any figure in this post.
What we ran
Two pieces of benchmark vocabulary, repeated from Part 1 because everything below depends on them:
- Rubric. The list of pass or fail criteria attached to a task. An AI judge reads the agent's output and checks it against each criterion. In Harvey LAB the scoring is all-pass, so a task scores 1.0 only if every criterion passes.
- Arm. One of the two setups being compared in a controlled test. Everything is held identical between them except the single thing under test. Here that single thing is the document tooling.
Part 1 covered what the two arms hand back: half the baseline redlines came back with their applied formatting stripped, and the grader never noticed. This post is the other half of the promise we made there. It is about what each arm costs to get there.
Harvey LAB holds roughly 1,760 tasks. We selected the 215 where redlining is the core work, which is a structural filter rather than a judgement call: a task qualifies if its deliverable is a redlined .docx, or if its input documents physically contain tracked changes. That is 55 tasks that edit documents and 160 that analyse marked-up ones, across 23 practice areas. We ran all of them, not a sample.
Setup. 215 tasks, run twice, 430 runs total, one run per task per arm, all completed and scored. Agentgemini-3.6-flashat high reasoning in both arms. Judgegemini-3.6-flashwith--docx-track-changes all, so tracked insertions, deletions and comments are visible to the grader on both sides. 11,997 rubric criteria judged per arm, averaging 56 criteria per task. Adeu version 1.31.0. Total provider-billed spend: $304.95 for the baseline arm, $256.16 for the Adeu arm.
The two arms:
- Baseline. Harvey LAB exactly as it ships. The agent reads and edits
.docxfiles with general-purpose tools: a shell, Python, and the document scripts bundled with the benchmark. This arm is byte-identical to upstream LAB. - Adeu. The same harness, the same model, the same grading, with the document layer replaced by our redlining engine. Four tools: extract, diff, apply edits, accept-all. LAB's own redlining scripts are removed from this arm. Nothing else changes.
What this comparison is, and what it is not. The baseline arm uses the reference document scripts published in Harvey's open-source benchmark repository. It is not Harvey's commercial product, and we have not tested Harvey's commercial product. Nothing here tells you how Harvey redlines a contract for a paying customer, and none of it should be read that way.
What it cost
| Metric, per task | Baseline | Adeu | Change | Adeu wins | Sign test |
|---|---|---|---|---|---|
| Cost, provider-billed | $1.42 mean / $1.25 median | $1.19 mean / $1.01 median | −16% mean / −19% median | 140 of 215 | p = 1.1×10⁻⁵ |
| Wall-clock time | 5 min 17 s mean / 4 min 44 s median | 4 min 21 s mean / 3 min 50 s median | −18% mean / −19% median | 152 of 215 | p = 1.1×10⁻⁹ |
| Agent turns, mean | 32.2 | 29.1 | −10% | 114 of 215 | p = 0.107, not significant |
| Total tokens, mean | 2.81M | 2.71M | −4% | 108 of 215 | p = 1.0, a wash |
| Rubric criteria passed, pooled | 92.4% | 93.4% | +1.0 point | 84 to 72 by task | p = 0.38, not significant |
The quality row is the one that makes the rest of the table mean anything. Grading is the model's job and the model is the same on both sides, so the two arms passed rubric checks at 92.4% and 93.4%, a difference well inside noise. We are not claiming Adeu produces better answers, and the data would not support it if we did. What the numbers support is narrower: the same work, done to the same standard, for less money and in less time.
Cost here is what the provider actually billed. That required correcting how LAB accounts for cached tokens, which is a change worth understanding before you weigh the cost figures, so it is set out in full below under "What we changed in Harvey's LAB", counterfactual included. The wall-clock result involves no pricing model at all.
Averages hide outliers, so here is every task individually. 132 of 215 (61%) were both cheaper and faster. 88 of 215 (41%) were cheaper, faster and scored no worse on the rubric. 55 tasks were both costlier and slower, so the swap is not free everywhere, and we come back to where those cluster below.
Where the money goes
Total tokens are a wash, down 4% with a sign test of p = 1.0. So why is the bill 16% lower? Because the token mix changed, and the mix is what you pay for.
| Component | Rate | Baseline | Adeu | Change |
|---|---|---|---|---|
| Fresh input tokens | $1.50/M | $0.58 | $0.51 | −$0.07 |
| Cached input tokens | $0.15/M | $0.36 | $0.35 | −$0.01 |
| Output and thinking tokens | $7.50/M | $0.48 | $0.33 | −$0.15 |
| Total | $1.42 | $1.19 | −$0.23 |
About two-thirds of the saving is generation. Output and thinking tokens per task fell from 64,435 to 43,790, a 32% drop, and the Adeu arm wins that comparison on 195 of 215 pairs at p = 3.1×10⁻³⁷. That is the most lopsided statistic in the dataset by a wide margin. Output tokens alone fell 42%, from 38.5k to 22.2k. Thinking tokens fell 17%, from 25.9k to 21.6k. Input tokens differ between the arms by only 3%.
This is also most of the time story. Generation is the slow part of a model call, and the Adeu arm does a third less of it, at 8.9 seconds per turn against 9.8. The saving is not that the agent read less. It is that the agent wrote less.
Both arms cache heavily and near-identically, with 86.2% and 87.3% of input tokens served from Gemini's implicit prompt cache, giving effective input prices of $0.34 and $0.32 per million. The cache does not favour either arm. It just moves the money to where the arms actually differ.
What the model types
We counted every character of tool-call arguments the model wrote across all 430 transcripts. This is the agent's keystrokes: shell commands, Python programs, edit payloads, memo text.
| Per task | Baseline | Adeu |
|---|---|---|
| Total characters typed | 128.0k | 75.1k, 41% fewer |
| of which, reading documents | 59.3k, hand-written python-docx programs | 2.2k, tool arguments |
| applying edits | 26.8k, driving LAB's redline scripts | 29.3k, the JSON edit payloads |
| writing memos and other files | 39.2k | 40.5k |
To look inside .docx files, the baseline agent wrote 2,931 python-docx programs, each of whose printouts, around 5,000 tokens on average, then flowed back into its context. The Adeu agent issued 2,072 extract and diff calls averaging about 80 typed characters each. Twenty-seven times less typing for the same visibility.
Note the edit row, which is roughly equal. The edit payload, meaning the actual text to change, is irreducible. What a dedicated tool removes is the program wrapped around the payload.
| Action | Mechanism | Calls | Tokens returned per call | Characters typed per call |
|---|---|---|---|---|
Read a .docx | baseline: bash and python-docx | 2,931 | 5,043 | 4,350 |
Read a .docx | Adeu: adeu_extract | 1,926 | 6,279 | 81 |
| Enumerate all changes | Adeu: adeu_diff | 146 | 18,523 | 131 |
| Apply tracked edits | baseline: LAB redline scripts | 960 | 1,337 | 5,038 |
| Apply tracked edits | Adeu: adeu_apply_edits | 1,163 | 2,419 | 4,677 |
Adeu's apply is transactional, so a failed batch writes nothing. Of 1,163 apply rounds, 673 landed clean, 185 partially landed through the wrapper's auto-salvage, which reports the failures, applies the valid edits and costs no extra turns, and 305 wrote nothing and reported why. That is a 74% land rate per round with byte-safe failures. Tool-failure events per run were 2.29 in the baseline arm and 2.04 in the Adeu arm. Neither arm ever emitted parallel tool calls, in zero of roughly 13,000 turns, which is a model behaviour and uniform across both.
The files themselves
Part 1 showed that LAB's bundled redlining script returns documents with their applied formatting destroyed, and that no rubric notices. That was measured on an earlier 36-task set. It replicates on this corpus, on the same runs behind every number above.
The task manifest itself names 52 deliverables per arm as redlined versions of a specific source document. Comparing each against its source in the raw OOXML:
| Baseline | Adeu | |
|---|---|---|
| Redlines with applied formatting stripped | 26 of 52 (50%) | 0 of 52 |
| Lost every table, of 42 sources with tables | 19 | 0 |
| Median formatting retention against source | 9.8% | 124%, lowest single file 102% |
Adeu's median is above 100% because its own tracked insertions add formatted runs. All 52 baseline runs invoked LAB's redline.py, and all 26 flattened files came from it. The 50% flatten rate matches Part 1's 16 of 32 within a point: same mechanism, larger sample, fresh runs. The method is in Part 1 and we will not re-argue it here.
The short version: the redline that costs 16% less is also the one you can actually send, with headings still bold and fee schedules still tables.
What we changed in Harvey's LAB
Every change below is symmetric across the arms except the experiment itself.
The experiment. In the Adeu arm, LAB's document scripts (redline.py, comments_add.py, accept_changes.py, and the XML pack and unpack helpers) are removed and four Adeu tools are registered instead, with the docx skill manual replaced by an authoring-only variant plus an Adeu manual. The baseline arm is byte-identical to upstream LAB: prompts, tools, scripts, agent loop and scoring untouched.
Cost accounting. We changed LAB's pricing to match the provider's actual bill, and this is the change most worth scrutinising. Upstream LAB prices every input token at the fresh rate, $1.50 per million for this model. In reality an agent loop re-sends its conversation every turn, and Gemini serves repeated prefixes from its implicit prompt cache at $0.15 per million, a tenfold discount that upstream accounting ignores. It also drops thinking tokens, which Gemini bills at the output rate but reports separately, and Pro-tier context-threshold pricing. We recorded per-request cached, thinking and tier token counts in both arms and priced runs accordingly. Both arms were swept fresh under identical accounting, with no old runs re-priced and no mixing.
Under upstream's flat pricing the same runs still favour Adeu, but the gap is smaller and not statistically significant, at −5.7% mean and −10.7% median, winning 112 to 103, p = 0.59. Adding thinking tokens gives −6.1%, winning 116 to 99, p = 0.28. Flat pricing overstates the real bill by more than three times, because it charges cached tokens at the fresh rate. That inflates input until it accounts for around nine-tenths of the total, which buries the generation gap, and generation is where the actual physical difference between the arms lives. The wall-clock result of −18% at p = 1.1×10⁻⁹ involves no pricing model at all and points the same way. The correction does not create Adeu's win. It stops mispricing from hiding a real one.
Two of our three accounting corrections, on thinking tokens and Pro-tier splits, *raise* reported costs. Only the cache correction lowers them. All three move reported cost towards the provider's real invoice.
Symmetric fixes made along the way, each applying identically to both arms: judging runs with --track-changes all so deletions and comments actually reach the grader; a comments-part rename so the converter shows both arms' margin comments, since pandoc only reads comments.xml while Adeu writes the equally valid comments1.xml, and leaving this unfixed would have biased the result *against* Adeu; retry and backoff hardening for rate limits; and crash guards for a Gemini SDK edge case. Judge, rubric, scoring logic and agent loop are otherwise upstream.
What we did not do. No baseline prompt or tool changes. No re-runs of unfavourable results: one run per task per arm, all 430 reported. No staging route for edits by file, which we removed early as un-benchmark-like. No author-name spoofing to bypass Adeu's multi-author guard.
Where the win concentrates
| Category | Tasks | Cost | Cost wins | Time | Turns |
|---|---|---|---|---|---|
| Produce a redline | 55 | −37% | 46 to 9 | −30% | −21% |
| Analyse markup | 160 | −8% | 94 to 66 | −13% | −5% |
A dedicated editor earns its keep where editing is the work, and 76% of those tasks came out both cheaper and faster. Review tasks still favour Adeu but modestly, because reading is a smaller share of their spend and both arms ultimately hand the same text to the same model. The 55 pairs that came out costlier and slower sit almost entirely on the review side, 47 of them, where 29% of pairs regressed on both metrics against 15% of the tasks that produce a redline.
On distribution: medians beat means in both headline metrics, at −19% each, and the tails move too, with p90 cost falling from $2.21 to $1.96 and p90 time from 7 min 44 s to 6 min 47 s. The single worst run in the corpus is ours. One first-turn redline of a master services agreement cost $8.75 against its baseline twin's $2.34, and took 29 minutes against 8, after a long edit-retry loop. It also passed 100% of its criteria, against the baseline twin's 97%. One run per task means individual pairs are noisy, which is why the win rates and medians are the readings to trust.
What this does not show
- Quality, in either direction. 92.4% against 93.4% of rubric criteria is parity, not a result. Same model both arms.
- The all-pass metric. LAB's harshest reading, tasks passing *every* criterion, was 40 of 215 for the baseline against 32 of 215 for Adeu, which is 18.6% against 14.9%, sign test p = 0.22, not significant. With around 56 criteria per task and one run per task, all-pass is a coin-flip-sensitive ceiling metric, and it measures the model rather than the tools.
- Fewer steps. Turns fell 10% but at p = 0.107. That is directional, not significant, and we are not claiming it.
- Fewer tokens overall. Total tokens are a wash at −4%, p = 1.0. The win is in the token mix, not the volume.
- Anything about frontier models. Everything here is
gemini-3.6-flash. Running 430 agent runs plus roughly 24,000 rubric judgments on a frontier model was out of budget. A cheap-model A/B favours neither arm, since the model is identical on both sides, and tool efficiency matters most in exactly the high-volume tier where legal teams would deploy at scale. Earlier 30-task pilots on other models pointed the same way, but we are not presenting those as evidence. - Absolute pass rates. The judge is the same model family as the agent. Self-preference exists, and it applies equally to both arms, so arm-against-arm comparison is reasonably protected. Absolute quality is not, and we never quote it as such.
- Harvey's commercial product, which we have not tested and which is not what LAB is.
Check the numbers yourself
Every figure in this post is derived from one JSON file, published under CC BY 4.0. It carries all 215 per-pair records with cost, wall-clock time and criterion rates for both arms, the aggregate statistics under all three pricing rules including LAB's original one, the bill decomposition, cache behaviour, the typed-character analysis, per-practice-area and per-category splits, and the 52-document fidelity audit with per-file OOXML counts.
The redlining engine is MIT licensed and public at github.com/dealfluence/adeu. The benchmark is public at github.com/harveyai/harvey-labs. Both engines are open source, so this is a comparison you can run yourself.
The document layer is a line item
There is a habit in agent design of treating the document layer as plumbing: give the model a shell and a scripting language and let it work the problem out. It does work the problem out. The 92.4% and 93.4% rubric rates say so. But it works it out by writing thousands of throwaway programs, and generated tokens are the most expensive and slowest thing in the loop.
Give the same model a tool built for the job and the programs disappear. The bill falls 16%, the clock falls 18%, and, as Part 1 showed on these same runs, the documents that come back are ones you could actually send.
Same model, same tasks, same grader. The tooling is not plumbing. It is a line item.
Adeu is open source: github.com/dealfluence/adeu
Citation
Harvey LAB is MIT licensed, and Harvey asks that it be cited in research that uses it. This post does, so for the record:
Harvey AI. Harvey LAB: The Legal Agent Benchmark, version 1.0, 2026. Repository at github.com/harveyai/harvey-labs, announced in Introducing Harvey's Legal Agent Benchmark.
The engine from this study is open source
The redlining engine in the second arm is MIT licensed and free to embed. Read the code, run the benchmark, check our numbers.
Explore the engine