![\"\"](\"https:\/\/abovethelaw.com\/wp-content\/uploads\/sites\/4\/2017\/07\/legal-research-on-laptop-computer-300x200.jpg\")
Does the legal research platform you use matter to the results you receive? Turns out, it very much matters. Different legal research platforms deliver surprisingly different results. In fact, in a comparison of six leading research providers, there was hardly any overlap in the cases that appeared in the top-10 results returned by each database.<\/p>\n
This startling finding is the result of research performed by Susan Nevelow Mart<\/a>, director of the law library and associate professor at the University of Colorado Law School, where she teaches advanced legal research and analysis and environmental legal research. Mart has published a draft of her research paper, The Algorithm as a Human Artifact: Implications for Legal {Re} Search<\/a>, and she presented some of her findings in a program I attended at the recent annual meeting of the American Association of Law Libraries<\/a>.<\/p>\n Mart\u2019s exploration of the differences among research services was spurred in part by an email she received from Mike Dahn, senior vice president for Westlaw product management at Thomson Reuters, in which he noted that \u201call of our algorithms are created by humans.\u201d Why is that statement significant? Because if search algorithms are built by humans, \u00a0then those humans made choices about how the algorithm would work. And those choices, Mart says, become the biases and assumptions that get coded into each system and that have implications for the results they deliver.<\/p>\n So Mart set out to study how hidden biases and assumptions affect the results provided by some of the major legal research providers. She chose six to study: Casetext, Fastcase, Google Scholar, Lexis Advance, Ravel and Westlaw. The results, Mart writes, \u201care a remarkable testament to the variability of human problem solving.\u201d<\/p>\n Little Overlap in Results<\/strong><\/p>\n “Remarkable” is the right word. Mart found that there is hardly any overlap in the cases that appear in the top 10 results returned by each database. An average of 40 percent of the cases were unique to one database, and only about 7 percent of the cases were returned in search results in all six databases.<\/p>\n This startled me. Of course I would have expected there to be differences among the top results delivered by different research platforms. But for there to be \u201chardly any overlap\u201d? I mean, the most relevant cases are the most relevant cases, aren\u2019t they?<\/p>\n Mart first compared the algorithms of different legal research services in 2013, when she entered the same query into four databases \u2013 Lexis Advance, Fastcase, WestlawNext and Google Scholar \u2013 and examined the results. That study produced three notable takeaways:<\/p>\n In 2016, she repeated the same test against the same four databases. Again she found very different results among the four. In addition, she found that each service returned notably different results than it had three years earlier, even though many of the new cases existed in the database when the first search was performed.<\/p>\n Expanding Her Scope<\/strong><\/p>\n For her latest research, Mart expanded her experiment to include 50 different searches in six research databases, and then to examine only the top 10 results. In order to makes the sets of information being searched as identical as possible, searches were limited to subsets of reported cases within specific jurisdictions.<\/p>\n One hypothesis Mart wanted to test was whether each algorithm would find the same cases. After all,\u00a0the search algorithm for each database was trying to achieve the same result in the same pool of information by finding relevant cases. Even though her earlier studies using one query had not found this to be true, there was the possibility that a large number of searches would produce more consistent results.<\/span><\/p>\n To the contrary, the different algorithms found very different cases. In fact, each research platform returned an average of 40 percent unique cases. On average, 25 percent of the cases were in only two databases, 15 percent appeared in three databases, 9.5 percent appeared in four databases, just under 7 percent appeared in five databases, and just under 7 percent appeared in all six databases.<\/p>\n Another hypothesis was whether, because the algorithms all rank relevance, and the goal is to return relevant cases, the top 10 cases would all be relevant. Here again, Mart\u2019s study found otherwise. The more-established research providers, Westlaw and Lexis Advance, did better at delivering relevant cases, but no service was perfect. Here are the percentages of relevant cases they delivered:<\/p>\n One further factor Mart examined was the number of unique cases delivered by each platform that were also relevant. In other words, given that each platform returned an average of 40 percent unique cases, how many of those unique cases were relevant? Westlaw did the best, she found, with 33.1 percent of its cases both unique and relevant. Casetext fared the worst, with just 8.2 percent of its unique cases also found to be relevant.<\/p>\n What It All Means<\/strong><\/p>\n In her research paper, Mart examines other differences in the results among research services. For example, newer services tended to deliver newer cases. She also discusses some of the factors that may explain, at least in part, why those differences occurred. But the bigger question for the rest of us is what we should make of this. Mart has several suggestions.<\/p>\n For one, we as researchers should have at least some understanding of the biases inherent in different systems and should not limit our research to any single system.<\/p>\n \u201cLegal research has always been an endeavor that required redundancy in searching; one resource does not usually provide a full answer, just as one search will not provide every necessary result,\u201d Mart writes. \u201cThis study clearly demonstrates that the need for redundancy in searches and resources has not faded with the rise of the algorithm.\u201d<\/p>\n For another, Mart believes we should all challenge legal research companies to be much more transparent about the biases in their algorithms.<\/p>\n \u201cAlgorithmic accountability in legal databases will help assure researchers of the reliability of their search results and will allow researchers greater flexibility in mining the rich information in legal databases,\u201d Mart argues. \u201cIf researchers know generally what a search algorithm is privileging in its results, they will be better researchers.\u201d<\/p>\n Two research platforms that I am aware of provide some degree of transparency into their search algorithms by allowing users to customize the relevance factors used by their algorithm. In the desktop version of\u00a0CaseFinder<\/a>, a Virginia legal research service, users can adjust sliders corresponding to four relevance-ranking factors: hit density, age, precedence, and enhanced precedence.<\/p>\n Recently, Fastcase<\/a> added a similar feature to its advanced search in Fastcase 7, allowing users to customize the algorithm by adjusting sliders for seven relevance factors: search relevance score, large document relevance, small document relevance, authoritativeness, frequently read, frequently printed, and frequently emailed.<\/p>\n The bottom line is to be cognizant of the fact that different research services have different biases and that those biases affect the search results they deliver. We can push companies to be more transparent about their algorithms, but in the meantime, we should remember that time-worn piece of advice: Consider the source.<\/p>\n In a comparison of six leading research providers, there was hardly any overlap in the top 10 results from searches.<\/p>\n","protected":false},"author":18763,"featured_media":471700,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[740,11],"tags":[725,474,726,44,728,327,729,730,731,689,732,691,733,734,735,736,66,737,7,738,33,739],"class_list":["post-1114","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-legal-research","category-technology","tag-aall","tag-algorithms","tag-american-association-of-law-libraries","tag-bob-ambrogi","tag-casefinder","tag-casetext","tag-fastcase","tag-fastcase-7","tag-google-scholar","tag-legal-research","tag-lexis-advance","tag-lexisnexis","tag-lexisnexis-lexis-nexis","tag-mike-dahn","tag-ravel","tag-ravel-law","tag-robert-ambrogi","tag-susan-nevelow-mart","tag-technology","tag-thomson-reuters-westlaw","tag-westlaw","tag-westlawnext"],"acf":[],"yoast_head":"\n\n
\n
\n![\"Robert](\"https:\/\/abovethelaw.com\/wp-content\/uploads\/sites\/4\/2016\/06\/Robert-Ambrogi-Bob-Ambrogi-150x150.jpg\")
Robert Ambrogi<\/a>\u00a0is a\u00a0Massachusetts lawyer<\/a>\u00a0and journalist who has been covering legal technology and the web for more than 20 years, primarily through his blog\u00a0LawSites.com<\/a>. Former editor-in-chief of several legal newspapers, he is a fellow of the\u00a0College of Law Practice Management<\/a>\u00a0and an inaugural\u00a0Fastcase 50<\/a>\u00a0honoree. He can be reached by email at\u00a0ambrogi@gmail.com<\/a>, and you can follow him on\u00a0Twitter<\/a>\u00a0(@BobAmbrogi).<\/em><\/strong><\/p>\n","protected":false},"excerpt":{"rendered":"