Conditions For AI Success: Discipline, Data, And Patience

At both AALL and ILTA this past summer, I attended sessions where Brian Kuhn from IBM spoke about transforming the business of law using IBM’s Watson offering. Brian offered a compelling vision of the future and when talking about Watson, while he did reference three use cases for Watson in the legal market, I did find it curious how much emphasis was devoted to Watson’s success in the medical field. Given the degree of hype currently surrounding AI in the legal markets, I thought it relevant to examine this in a bit more detail. Two particular questions came to mind:

1. What is the track record for applying AI in medicine?
2. What learnings from the same are relevant when looking at the potential impact of AI on the legal market?

Despite the high degree of irrational exuberance (to paraphrase Alan Greenspan and Robert Shiller) around artificial intelligence, the reality is that applying AI to complex problems is, at best, a nuanced proposition and one that typically entails more investment than one may think. One highly publicized example can be found in IBM’s agreement with the University of Texas MD Anderson Cancer Center. Widely regarded as one the world’s leading cancer treatment facilities, MD Anderson announced in 2013 that it would partner with Watson on a “moonshot” effort aimed at eradicating cancer. While launched with significant optimism and publicity, the project ran into multiple problems and was shut down after three years and an estimated $68M in expenses (not counting the soft cost of the considerable time spent by physicians, IT, and administrative staff as well as investments in technology infrastructure). Despite the hype, Watson failed to diagnose even one patient in a clinical setting.

In another example, IBM partnered with Sloan Kettering to build Watson for Oncology with the promise that the system would ingest medical data and articles in order to ultimately identify “new approaches” to cancer care. In this case, Watson has been used by physicians clinically, but a recent damning report by STAT shows that the actual performance falls far short of the lofty marketing statements. I found one cancer specialist’s comment particularly chilling. He stated, “Watson for Oncology is in their toddler stage, and we have to wait and actively engage, hoping to help them grow healthy.” This is after six years and an unknown (but very large) expenditure of time and financial resources.

While it may be tempting to bash IBM (and Watson) based on these examples, I think it is more instructive to ask what we can learn from each of them. The availability of resources and capability (both subject matter and technology expertise) were clearly not the problem in either case. My sense is that the projects themselves may have been doomed from the start because the essential conditions for success were not in place. From my perspective, there are three such necessary conditions for the application of AI technology to any knowledge profession.

1. A tight, well-defined use case.

The term use case starts to venture into the land of “tech speak,” but for the purposes of this discussion, we can consider a use case to be a problem that has a defined starting point and expected outcome. If one can easily describe both the starting point and the expected outcome, that use case is considered to be “tight” and much easier to both build and verify. Taking an example from the legal markets, document review as part of due diligence in an M&A transaction would be such a use case: an attorney wants to identify contracts where certain conditions exist (e.g., supplier contracts that are not assignable on change of ownership). A technology provider can easily look at that use case and define the results that represent success.

On the other hand, take the example of contract review as part of drafting and negotiation: an attorney wants to review proposed language and modify terms to improve the client’s outcome and lower the risk profile. That is a much more nuanced and subjective task, and one which a technology provider would have difficulty in defining the results that represent success.

In general, the best practice at this time is to stay away (far, far away) from broad, loosely defined use cases.

2. Make sure you have enough consistent data.

Since algorithms can process much more data much, much faster than humans, data is naturally at the heart of AI-fueled application. In order to train a machine to identify specific conditions, one needs to have a sufficient data set to use in training. The definition of “sufficient” depends on the difficulty of the problem (e.g., training a program to identify Governing Law is probably easier than training it to identify the terms of an Indemnity Clause), but hundreds of pieces of sample data will  support a tight use case — and significantly more are needed for a broad, loose use case.

As we’ve discussed previously, that data also needs to be consistent. If one were to train an algorithm to recognize terms in a stock purchase agreement in the U.S., and then attempt to apply that same algorithm for U.K. agreements, the algorithm will fail to perform well. Before embarking on an AI project, it is critical to ask yourself a few questions.

• Do you have enough (and the right) data?
• What format is that data in?
• Does the data require additional processing to be useful for the algorithm?
• What is the level of effort (in terms of cost, time, and resources) to get the data to the correct starting point for AI processing?

3. Make sure you have sufficient SME resources available

Finally, even if you have a tight use case and enough data, you’re going to need access to Subject Matter Experts (SMEs) to train the algorithm. The algorithm itself of course knows nothing about the subject at hand — be it law, m­edicine, or any other discipline. In my experience, this is the requirement that would-be AI implementers most often underestimate. Taking the examples above, in both cases the solutions required significant physician time before providing any return on investment. If one projects the same requirements into the law firm market, it could mean speculatively devoting otherwise billable resources.

You’ll notice that, of the three immutable conditions referenced above, none refers to technology. While having proper AI technology is indeed critical to success, we’ve happily reached a point where multiple vendors (including IBM) provide access to algorithms capable of supporting AI solutions for the legal market. Differentiation lies in how one applies that technology and in setting the conditions for success-that in turn will drive which solutions are able to rise above the hype.

---

Dean Sonderegger is Vice President & General Manager, Legal Markets and Innovation at Wolters Kluwer Legal & Regulatory U.S., a leading provider of information, business intelligence, regulatory and legal workflow solutions. Dean has more than two decades of experience at the cutting edge of technology across industries. He can be reached at Dean.Sonderegger@wolterskluwer.com.