How Will You Measure Your AI?

The late Harvard professor Clayton Christensen, best known for his breakthrough book “The Innovator’s Dilemma,” published another in 2010 entitled “How Will You Measure Your Life?” That book was based in part on the observation that many successful businesspeople from his graduating class left behind them a trail of broken personal relationships. One of his points was that “not a single one of them graduated with the deliberate strategy of getting divorced and raising children who would become estranged from them. And yet a shocking number of them implemented that strategy.”

Observations like that led him to write the book to help driven executives manage and balance multiple dimensions of their lives, as opposed to the dimension related to business alone.

2023 was a year of hype and excitement related to Generative AI (GenAI). Knowledge Management and Library professionals at law firms have been busy evaluating technologies from vendors and experimenting with the technology, but the pace has been unsustainable. Never before have managing partners been so interested in technology, and even clients are expressing interest. Some corporations ask out of caution, but others are interested to see if the new technology can help reduce their spend with outside firms.

GenAI does provide tremendous promise for the practice of law, and it certainly is a disruptive technology.  The next couple of years, though, will be years when its applications will need to prove a return on investment and deliver value.

Here are a few tips that can help in developing key performance indicators (KPIs) to measure and direct GenAI initiatives.

Experiment!

Most professionals in the legal industry are still in the early days of using GenAI and are still discovering what capabilities are out there, which ones show promise, and which ones don’t.

When evaluating opportunities, use the principles of experimentation and set forth a hypothesis.

Here is an example: Using GenAI to summarize a collection of dozens of filings can be done more effectively, efficiently, and in less time than an attorney doing the same task alone.

With a hypothesis, legal professionals can then design an experiment (aka a short project) to test out the hypothesis. When completed, the results can be reported out to see if the hypothesis can be proven or disproven. If the hypothesis can be proven, then it may be OK to move forward and perform another experiment related to the opportunity.

If the hypothesis is disproven, then pause and reflect. Perhaps there is another approach that will yield better results. If there aren’t any actionable ideas, then stop.

Be Willing To Stop

For many organizations, the effort it takes to overcome inertia to start a project is so great that once initiated, it gains momentum and can take on a life of its own.  It is critical to have discipline and processes in place to stop work that is going nowhere.

Stopping is hard for organizations. It implies failure. There is no right or wrong answer in experimentation, so it can remove the stigma of failure.

Using the techniques of experimentation can help adjust your culture and can position an organization to have true agility, to pivot, and to find the opportunities with real promise.

Prioritize Problems And Opportunities

As your experimentation and learning begins to show promise, begin to develop a list of problems to be solved and opportunities to pursue. These should align with the goals of your organization. Here are some examples:

• Grow revenue
• Decrease expenses
• Increase profitability
• Be more efficient
• Increase quality
• Reduce errors
• Increase customer/client satisfaction
• Acquire new customers/clients
• Strengthen the organization’s “brand”
• Be more competitive

By lining up prioritizing and aligning with the goals of your organization, it becomes easier to say “yes” or “no” with less subjectivity. Sometimes, you have to take on work for no other reason than the big boss wants it done.

But just because someone senior asks for something to be researched, that doesn’t always mean it is a direct order and the highest priority — or that they even want it done. Organizing and prioritizing can help present tradeoffs and provide clarification to focus resources and energy.

Establish Metrics And KPIs

For corporate law departments, establishing metrics can be relatively straightforward. Corporations are much more interested in the bottom line. Is something better? Faster? Less expensive?

There are more measurables in a law department. Measuring the reduction of time or expense in a task can be coupled with simple math to demonstrate a return on investment for the department.

For law firms, understanding the right metrics is more nuanced. Making a task more efficient doesn’t necessarily translate into more revenue or profit for the firm. It may actually reduce profitability. This is one reason why some firms struggle with embracing technology.

For firms, GenAI initiatives may create some obvious opportunities, like eliminating nonbillable work. But that isn’t going to get many thank-you notes from partners unless it increases billable hours instead.

Law firms may need to focus a little more on the bottom half of the list above. Coming up with productivity improvements may need to be measured from the perspective of staying competitive in a changing environment or for helping acquire new clients.

Just as Clayton Christensen observed about the personal lives of his peers, nobody intentionally sets out to fail — and yet many implement that strategy. By experimenting and pivoting, by prioritizing to be more proactive to focus on what is actually important, and by developing metrics, one can better ensure positive results from GenAI initiatives.

And with the unsustainable pace of all that is being thrown upon legal innovators at the epicenter of evaluating GenAI, perhaps the application of metrics to measure AI will also help in providing a little more work-life balance, too.

---

Ken Crutchfield is Vice President and General Manager of Legal Markets at Wolters Kluwer Legal & Regulatory U.S., a leading provider of information, business intelligence, regulatory and legal workflow solutions. Ken has more than three decades of experience as a leader in information and software solutions across industries. He can be reached at [email protected].