We set out to determine with frontier AI model performed best on M&A due diligence. We set out to test this by creating a synthetic data room, and asking each AI model to find red flags and risks across the data room.
Experiment design
We created a synthetic data room with 153 files in it, and we embedded 48 planted risks. Then we tasked each frontier AI model to act as a private equity associate running due diligence to flag potential risks.
For each frontier AI model, we tracked:
- How many of the planted risks were identified
- How many findings did each AI hallucinate and invent (findings that were false)
- Did the model read all of the documents or only do a partial read?
The prompt
We used a prompt assigning the model to act as a PE associate running a due diligence on a deal. The model was responsible for identifying which potential risks to look for and finding those risks. Every model was given the same prompt.
Scoring
We analyzed the findings of each model against our list of planted risks. Each planted risk was then marked as found, partially found (indicating it was flagged but missing key information), or missed. We did not include partially found items in the metric scoring the percentage of items found.
Thinking levels
Some frontier models, like Claude, allow you to set a “thinking level”. Higher thinking level tells the model to spend more compute reasoning through the room before it commits findings.
In this test, Claude Opus 5 (thinking level: high) vs Claude Opus 5 (thinking level: extra) are the same model with different thinking levels applied. All other variables stayed consistent throughout all tests.
Results
Claude Opus 5 (thinking level: extra) flagged 85% of planted risks, the highest score in the test. The other configurations ranged from 79% for Claude Opus 5.5 down to 40% for Meta Muse Spark.
Hallucinations for each model
| Model configuration | Number of invented findings | What the model actually read |
|---|---|---|
| Claude Opus 5 (thinking level: extra) | 1 false | Read every file |
| Claude Opus 5 (thinking level: high) | 0 | Read every file |
| Claude Opus 5.5 (thinking level: high) | 1 false | Read every file |
| GPT-6-Astra (high) | 0 | Selective file read |
| Grok 4.6 | 0 | Selective file read |
| Meta Muse Spark | 6 false items | Selective file read |
What this means for private equity investment teams
Thinking level made a large impact on the ability of AI to find risks in the data room
Claude Opus 5 (thinking level: extra) flagged 85% of planted risks. Claude Opus 5 (thinking level: high) flagged only 48%. This was with the same data room, same prompts, and same list of planted risks. Telling the AI to look harder improved results in a large way.
Reading strategy shows up in the score
AI that read all of the documents found more risks. Grok and Muse Spark scored the lowest in the test, but didn’t do a full pass on every document. This is relevant because AI can’t read every document when the data room exceeds their context limit.
Hallucination rate
Several of the test runs produced no hallucinated findings. We believe this is because the size of the data room is only 153 documents, so larger models can fit everything into their context window.
When analyzing larger data rooms, AI models begin to exceed their context window. This results in findings being missed, because AI can’t look at every document. Hallucination rate increase as models use more context.
We are producing benchmark studies on larger data rooms to analyze how accuracy scales when assessing 1000+ document VDRs. When assessing large number of files, it becomes more critical to build purpose-built diligence tooling to accommodate the constraints that LLMs run into, such as context limits. This is a key component of our AI due diligence tool at Fulcra AI.
Where Claude isn’t a good fit in due diligence
This was on a small data room of only 153 documents. The accuracy that AI can produce in due diligence is dependent on the amount of data that it analyzes. Here’s what this means for running M&A due diligence:
- Leading frontier models, such as Opus 5, have a context size of 1 million tokens. They can fit ~1500 pages of information into their working memory. This means that in a large data room, the AI cannot read every document.
- LLMs become less accurate as context length increases. Studies show that hallucination rate increases by ~10x when AI is given more context (Hallucination rates grow with context length). In the context of due diligence, this means that if you give the AI a large corpus of documents, it will find less risks and hallucinate more.
- Some of our work at Fulcra is building a platform that can help identify risks as data rooms scale to 500+ documents.