Head-To-Head Showdown Between AI-Driven Legal Research Tools

It definitely matters which AI tools you choose to use.

UPDATE: Check out below where Lexis outlines some issues they have with the quality of the results.

Everyone talks about “AI” or “machine learning” or any one of the numerous euphemisms for artificial-intelligence solutions. The language around the technology has softened from the height of its hype cycle, but there’s still a sense out there that AI is this “thing.” As one legal tech leader put it to me last year, “a lot of lawyers act like ‘we need to get some AI’ without trying to figure out how AI solutions might be different.”

To some extent, that still holds sway. It’s a conclusion that’s not entirely off base because some solutions use the same underlying AI algorithms. Still, even in those cases, the work that goes into building something atop that foundation matters. And when providers have wholly distinct code under the hood, the differences really matter.

Unfortunately, the legal sector doesn’t really have its own Consumer Reports — a bunch of folks running around pressure testing legal tech day in and day out. So when the National Legal Research Group conducted a study comparing Casetext’s CARA AI against LexisNexis to determine which system provides users with the fastest, best results, it filled an important gap in the sector.

Casetext, who would never shy away from head-to-head matchups, emerged victorious with the NLRG finding that attorneys complete their research 24.5 percent faster using Casetext CARA AI compared to LexisNexis — requiring on average 4.4 times fewer searches on CARA to get the right results. That may not sound like much, but the study figured that these time savings add up to between 132 to 210 hours a year — a number they arrived at based on ABA figures on the amount of legal research attorneys do every year.

And CARA wasn’t just faster. According to the report, the attorneys studied rated CARA’s results 20.8 percent more relevant than those they found through LexisNexis.

Casetext’s Jake Heller told me of the study that the real winners for his product are the small law firms and solo practitioners. “If you’re small, you have to find ways to make your job faster or you won’t have time to run your practice.” Picking up an extra 200 hours a year would seem like a pretty good exchange.

Sponsored

While these showdowns may seem antagonistic, they really aren’t. Figuring out where products perform in relation to each other is critically important to driving the next generation of updates. LexisNexis isn’t going to take these results lightly and they are undoubtedly already working on improving their system. Too bad we don’t see more comparisons like this.

UPDATE: LexisNexis outline some reasons why they don’t think the results paint a complete picture:

In response to an inquiry from LexisNexis, John Buckley, President of National Legal Research Group, responded: “Our participation in the study primarily involved providing attorneys as participants in a study that was initially designed by Casetext. We did not compile the results or prepare the report on the study—that was done by Casetext.” Nowhere is this relationship disclosed in the report paper nor is the report labeled as work-product of Casetext.

The fact that Casetext pitched the idea to a company that does legal research isn’t tremendously troubling, though it should be noted. A bigger concern would be exactly how much training and familiarity the attorneys had with each system before the study began. Lexis argues that:

NLRG participants were also ‘trained’ in the use of Casetext prior to the test. With only a brief introduction to Lexis Advance, it was presumed that all participants already had a basic familiarity with Lexis Advance and all of its AI-enabled search features.

Sponsored

That’s a hard thing to control for but definitely a concern with any study.

Just more reason why we need a dedicated Consumer Reports for these sorts of studies to keep consistent methodologies over time.


HeadshotJoe Patrice is a senior editor at Above the Law and co-host of Thinking Like A Lawyer. Feel free to email any tips, questions, or comments. Follow him on Twitter if you’re interested in law, politics, and a healthy dose of college sports news.

CRM Banner