{"id":30914,"date":"2024-10-22T15:15:47","date_gmt":"2024-10-22T19:15:47","guid":{"rendered":"https:\/\/abovethelaw.com\/?p=1076435"},"modified":"2024-10-22T15:15:47","modified_gmt":"2024-10-22T19:15:47","slug":"benchmarks-and-outcomes-moneyball-for-genai-part-i","status":"publish","type":"post","link":"https:\/\/abovethelaw.com\/legal-innovation-center\/2024\/10\/22\/benchmarks-and-outcomes-moneyball-for-genai-part-i\/","title":{"rendered":"Benchmarks And Outcomes — ‘Moneyball’ For GenAI (Part I)"},"content":{"rendered":"

\"moneyball\"It\u2019s October. The seasons are changing. The air is growing crisper. And people in the United States are beginning to take more interest in Major League Baseball as the World Series fast approaches. In celebration of the season, we invite legal professionals to revisit “Moneyball<\/a>,” the 2011 sports drama starring Brad Pitt and directed by Bennett Miller.<\/p>\n

Pitt plays Billy Beane, who, it should noted upfront, is neither a lawyer nor an AI expert. He is however the general manager of the 2002 Oakland A\u2019s, a lifelong student of the game who struggled to consistently produce a winning team in a small baseball market. Beane didn\u2019t have the budget to compete for players like teams in larger markets like New York or Boston. The A\u2019s could develop players, but they couldn\u2019t retain them when they became stars. They did, however, have access to the vital statistics.<\/p>\n

Baseball is a sport with a century of data behind it and benchmarks like a player\u2019s batting average are known by even casual fans. What Billy and the A\u2019s did was use analytics<\/a> and different key performance indicators to win. A player\u2019s batting average is a great metric, but it doesn\u2019t account for other factors like the ability to get on base so on-base percentage is better. Getting on base leads to more runs scored. And more runs scored means winning more games.<\/p>\n

When losing a star player like Jason Giambi to the New York Yankees or Johnny Damon to the Boston Red Sox, conventional wisdom would say the team would have to replace two stars. But Beane, recognized the A\u2019s needed to replace the production<\/em> of the players they lost. In aggregate, they needed to get on base as much as the prior year. Using analytics, the A\u2019s would go on to set an American League record by winning 20 straight games in a row and also made it to the World Series. But the way they did it was the bigger story.<\/p>\n

So what can \u201cMoneyball\u201d teach us about benchmarking AI in legal?<\/p>\n

The Stanford study on benchmarking GenAI solutions<\/a> this past summer moved the conversation forward regarding the usefulness and impact of GenAI solutions. The study was not without some controversy that also helped in generating awareness to an important topic: How do we measure the results of GenAI on the legal industry.<\/p>\n

The Stanford study tested leading research products on their ability to create answers to questions related to caselaw research. A correct answer was one that accurately reflected the current state of the law. An answer that did not reflect the current state of the law was considered a hallucination. The result? One in six queries hallucinated.<\/p>\n

The definition of hallucination in the study is great for benchmarking. But does a hallucination as defined in the study always equal a bad outcome? What if the answer moved your research in the right direction and then you were able to formulate a Boolean search that answered your question? Another run crossing home plate.<\/p>\n

And what about associates using traditional research solutions? Has anyone benchmarked their legal research skills to see how often their conclusions do not reflect the current state of the law?<\/p>\n

The key points are:<\/p>\n