What one Big Law firm is doing to eliminate ‘AI pilot fatigue’

• Law firms face an AI overload from testing too many software tools at once.
• At Cozen O’Connor, the firm has moved away from testing everything to a more disciplined approach.
• It makes a few bigger bets on tools it believes will teach it the most and deliver clear gains.

Law firms are getting hit with a blizzard of pitches from tech startups promising faster research, quicker drafting, and higher margins. For firm leaders, the challenge is no longer just finding useful tools. It’s stopping the deluge of AI software demos and trials from wasting time and energy.

At one Big Law firm, that problem has a name: pilot fatigue. Andrew Woolfe, Cozen O’Connor’s chief strategy and innovation officer, uses the term to describe what happens when organizations ask employees to test too many new products at once.

“Pilot fatigue is real,” Woolfe said in an interview with Business Insider. Cozen O’Connor, which has more than 1,000 lawyers worldwide and about $800 million in gross revenue, is a prime target for companies pitching AI-powered tools.

In an interview with Business Insider, Woolfe described how Cozen O’Connor has moved beyond that early wave of frenzied testing. Now, he said, the focus is less on running as many pilots as possible and more on setting priorities, choosing tools with real potential, and releasing them at scale when they prove useful.

For Woolfe, the issue took off a couple of years ago, when large language models made it easier for startups to create legal tech quickly and get it in front of law firms. Software companies, he said, offered attractive terms to land pilots, and firms — eager not to fall behind — said yes to a growing stack of tools.

The result was predictable. Lawyers, already overworked, were asked to spend still more time testing software.

Woolfe said Cozen O’Connor has become more disciplined about running pilots. Before testing a tool, the firm defines what success should look like. High usage could justify a wider rollout.

Sometimes the value is harder to put a number on. Woolfe said Cozen O’Connor also looks at whether an AI tool fits into real workflows, produces better results for clients, or makes lawyers happier at work. Attorney feedback, he said, is a key factor in deciding what the firm deploys.

Woolfe added that the firm tries to be thoughtful about who it pulls into pilots and how much it asks of them.

Typically, Cozen O’Connor favors tools that many lawyers across the firm could adopt, especially when a pilot can produce lessons that extend beyond a single practice group. But that doesn’t rule out specialized software. If a smaller group of attorneys can argue that an AI service would meaningfully improve how they work or benefit clients, he said, the firm is willing to take a closer look.

And a pilot can be worthwhile even if the software itself never sticks, Woolfe said. Each test helps the firm sharpen its ability to evaluate tools, train lawyers, and manage change. In his view, the key question is, “What’s going to help us develop institutional muscle around AI?”

To date, Cozen O’Connor has piloted AI software such as Laurel, which helps lawyers fill out their timesheets, DeepJudge, which lets law firms build custom search engines, and Harvey, which is designed for broad use across a law firm.

There’s no shortage of requests to evaluate new technologies, Woolfe said, from software companies and from lawyers inside the firm. But, he said, because time and budget are finite, “we need to make a few big bets that we’re going to learn from, that our attorneys are going to benefit from, and that our clients are going to benefit from.”

Have a tip? Contact this reporter via email at mrussell@businessinsider.com or Signal at @MeliaRussell.01. Use a personal email address and a non-work device; here’s our guide to sharing information securely.