Language Matters With AI And The Law

“You can’t put too much water in a nuclear reactor.”

That was the advice from a retiring power plant supervisor played by Ed Asner in a vintage “Saturday Night Live” skit. Asner’s character exits the plant for a final time, leaving behind a new team to debate whether his parting words meant they could put as much water as they wanted in the reactor or whether it meant don’t put in too much. Eventually, someone suggested that maybe the operative word was “you” and that it wasn’t their responsibility but the responsibility of the following shift. The skit ended with a mushroom cloud.

When it comes to the law, attorneys know that language matters. A misplaced comma or an improperly worded clause in a contract can create ambiguity or alter the meaning of a legal document. In court, the phrasing of a question or the response of a witness under cross examination is under scrutiny. Sometimes, it depends upon what the meaning of the word “is” is.

The recent Stanford Study on the accuracy of legal research services and the likelihood to hallucinate set the legal world abuzz. It focused on the ability for legal research services to provide error-free answers that reflected the current state of the law in caselaw research. There has been lively debate as some answers provided by systems may be helpful but include inaccuracies. Is that any different than an inexperienced associate researching with traditional tools? Different researchers might take different approaches and come up with different answers. The law isn’t always black and white. And in litigation, one party is looking for arguments in favor of their position, while opposing counsel is looking for the opposite.

The Stanford study is a leap forward as the conversation about measurement and benchmarking is progressing.

Output And Outcomes Matter

The output of systems is ultimately what the legal community will value. Do GenAI systems create better outcomes and can they be more accurate? All solutions created will need to continue to raise the bar. While an answer may not always be exact, there is also the broader context of how it contributes to the specific client, matter, and fact pattern. For more thoughts on measurements and prioritizing GenAI for business outcomes click here.

Input Matters Too

There is another part of the equation: the quality of input. Researchers do approach tasks differently. And a well-designed and high-functioning GenAI system is only going to be as good as the user’s input. Just as the supervisor retiring from the nuclear power plant’s advice was poorly worded, so too can an attorney’s prompting of a GenAI system.

It’s a striking contrast that attorneys who are trained to create an eloquent and persuasive argument or an ironclad clause in a contract can struggle when instructing a GenAI system. To be fair, significant behind-the-scenes preparation is needed to build that eloquent and persuasive argument. And GenAI technology is new, with user interfaces that have yet to unlock the creativity of attorneys when talking to a screen.

It also shouldn’t be surprising, though, that some Biglaw partners can be notorious for providing limited context — just ask associates. Partners are vital to firms and can bill out at hourly rates well north of $1,000. Their time is important. And just as a medical student’s internship is a proving ground to test the resolve of doctors, so is the environment in a Biglaw firm for an associate.  While it may be changing some, associates must prove themselves and demonstrate resourcefulness.

In this context a new associate might receive direction like, “Go to Standard Fed and tell me what the penalty is under code section 6662 if there is negligence involved. I need the answer now!”

The associate has to be resourceful and build a network of relationships to learn that Standard Fed is a publication. The associate likely knows the context is tax law, but they may not be familiar with code section 6662 and that it pertains to penalties for underpayment of tax for intercompany cross-border transactions. In the same way that associate is tested by limited context, so are GenAI systems. Perhaps GenAI will become as resourceful as human associates, learning more and gaining perspective with each new interaction.

How To Provide Better Input And Create Better Prompts

Precision in language matters and the more context provided, the better a machine (or a human) can perform. We recently conducted a prompting challenge with Suffolk University Law School because language matters and because we wanted to understand how new legal minds think about interacting with GenAI systems.

There are growing prompt libraries, and firms will continue to test and develop best practices for prompting. In a process known as “prompt injection,” systems can be designed to “inject” context into a prompt behind the scenes.

But what can attorneys themselves do? What about that seasoned partner that knows they will need to use GenAI?

Here is a simple approach. When approaching GenAI, think of prompting like you are mentoring an attorney you enjoy who has high potential. In those situations, individuals are patient and take time to explain things. Do the same when prompting GenAI. Take time to slow down. Explain the nuance of a legal concept and why you think it is important. Provide background and context just as you would if you are investing time in a less experienced attorney. Spell things out. Don’t abbreviate or use acronyms. Tell the GenAI system everything you want it to know and provide specific instructions.

The world of GenAI is changing every day. Anthropic just released a new version of Claude (Claude 3.5 Sonnet) that allows a user to input up to 200,000 tokens which translates to nearly 150,000 words or 300 pages of single-spaced material.  GPT-4o has similar capabilities, as does Google’s Gemini. The point here is that systems will continue to provide more abilities to command GenAI. It will be up to the systems to translate the inputs into quality output.

For the attorney users of GenAI, language matters. By providing more background and specific instructions, the retiring supervisor in the “Saturday Night Live” skit would likely have averted the comically dire ending of the power plant. Attorneys can improve the usefulness and outcomes related to the use of GenAI in the same way — even if the answers provided aren’t perfect.

---

Ken Crutchfield is Vice President and General Manager of Legal Markets at Wolters Kluwer Legal & Regulatory U.S., a leading provider of information, business intelligence, regulatory and legal workflow solutions. Ken has more than three decades of experience as a leader in information and software solutions across industries. He can be reached at [email protected].