Who’s Afraid of the Casetext Challenge?

When Pepsi introduced its “Pepsi Challenge” in the 1970s, Coca-Cola didn’t like it one bit.

For anyone unfamiliar, the Pepsi Challenge became a defining event in the cola wars of the 1970s and 1980s. Pepsi, then a scrappy regional brand, devised a simple marketing campaign. At malls around the country, it conducted blind taste tests between Pepsi and Coke.

The wildly successful ads that followed showed that more people choose Pepsi over Coke. Coke responded with a series of increasingly desperate measures, culminating in the disastrous launch of “New Coke” in 1985. Coke’s initial reaction, however, was to attack the Pepsi Challenge itself.

There are plenty of differences between the cola wars of yesterday and the competition among legal research tools today. That said, those of us at Casetext are getting a pretty good taste of what it was like to be Pepsi. Like Pepsi, we’re an upstart product. Like Pepsi, we’re working to unseat powerful incumbents. And like Pepsi, we’re eager to show how our product stacks up to the competition. Take this year’s Legalweek conference. After Sarah Glassmeyer tossed out a friendly challenge there, we staged a test between our A.I. search technology, CARA, and a much-hyped competitor. Casetext won handily; the other folks didn’t even show up to defend their product.

Last month we published a study measuring CARA A.I. against one of the most heavily entrenched legacy research services. In the study that Casetext commissioned, 20 attorneys from the National Legal Research Group performed research tasks using both CARA A.I. and the older brand, and answered questions about their experience. The findings of the study include the following:

• Three out of four attorneys preferred their research experience using CARA A.I. to the legacy provider.
• Attorneys finished their research nearly 25% faster using CARA A.I.
• Completing research tasks on the legacy research tool required 4.4 times as many searches as they did on CARA A.I.
• Attorneys rated their search results on CARA A.I. 20% more relevant than their results on the competitor.

Which brings us to another parallel between Casetext and Pepsi: we sure did piss off the old guard.

As with the Pepsi Challenge, the study has persuaded quite a few people that there are advantages to going with Casetext over the legacy providers. And as with the Pepsi Challenge, the giant that lost is getting defensive. Our competitor’s first reaction, like Coke’s, has been to attack the study. It has voiced three criticisms, which we’d like to address here.

First, they complain that our report “failed to fully disclose the relationship between Casetext and NLRG.” (We designed the research issues with input from NLRG, and also wrote up the results.) This is an odd objection, given how easy it is to refute. In a sponsored post Casetext published on ABA Journal, we characterized the study as “conducted by Casetext in conjunction with attorneys from the National Legal Research Group.” For its part, the industry blog Dewey B Strategic began its coverage by saying that “Today Casetext released a study it commissioned…” It didn’t take any investigative reporting to describe our role so accurately; we told Dewey B Strategic all about it.

Next, they complained that Casetext designed the study, and that our methods fell short of “those employed by independent lab study.” We don’t actually know what “independent lab study” methods are when it comes to legal research products; as far as we know, there have been no independent lab studies of the issues we explored in our study (though we’d love to see one). Regardless, when you’re Pepsi, you get to design the Pepsi Challenge. Of course, we invite them to identify any way in which our study was unfairly biased toward Casetext.  

Third, they bemoan that the researcher-participants in the study were given a 20-minute training on Casetext, but did not offer the same training on the legacy provider. The study assumed that legal researchers with an average of 25 years of practice experience would have some familiarity with the big, standard legal research technologies. If they truly do not believe that’s the case, it might be the most condemning result of all to emerge from the study.

There’s one final criticism we’d like to address – this one from Bob Ambrogi. Because we paid for our study ourselves, Bob analogizes us to “a tobacco company fund[ing] research on the health effects of smoking.” Of course, what we’re doing is different than tobacco. We build efficient, high-quality legal research that is accessible and affordable — not a product that kills people. There is no cover-up or conspiracy here. Beyond that, unlike tobacco, there is no independent third party pouring millions into studying comparing legal research products. If this kind of research is going to get done, it’s because challengers like us finance independent third parties. This is why others, like Joe Patrice, have recognized that our study has “filled an important gap in the sector.”  

Anyway, after it criticized the Pepsi Challenge, Coke confirmed the results of the test for itself. Only with its panicked introduction of New Coke did Coca-Cola stumble upon a successful strategy – one that activated the deep emotional attachment that consumers had to Coke. Maybe the big, old, legacy provider we’re competing against will be so lucky, but I doubt that many of their customers will choose not to switch to Casetext due to a deep emotional attachment to them.

We believe it will be the opposite. We’re a smaller, younger company going up against giants that have held near-monopolies on the legal research industry for decades. When you’re the smaller, newer challenger taking on goliaths, you need to work twice as hard — make a dramatically better and more innovative product; have affordable, transparent and simple pricing; and provide stellar customer service. That’s what we’re doing. It’s why they’re so scared of us.

Which might explain why our little study has gotten them so upset.

***

Jake Heller is the co-founder and CEO of Casetext. Before starting Casetext, Jake was a litigator at Ropes & Gray. He’s a Silicon Valley native, and has been programming since childhood. Want to join the Casetext challenge? Sign up for a free trial of CARA A.I. here.