Hallucinations, Inaccuracies, And The Erosion Of The Rule Of Law

Another week. Another law firm caught citing cases that didn’t exist and for incorrect propositions. All allegedly due to reliance on AI.

But this was not just any law firm. It was Sullivan & Cromwell, one of the largest and most influential law firms in the United States and, for that matter, the world. It’s emblematic of the threat of misplaced reliance on artificial intelligence not only to the lawyers and law firms but also to our own rule of law itself.

The Sullivan & Cromwell Error

The firm reportedly recently filed a motion in bankruptcy court that had numerous errors and incorrect citations. Once they were discovered, Sullivan & Cromwell of course immediately apologized to the court and even to the opposing law firm that discovered the mistakes. (It’s funny how apologetic you get once caught.)

And these were more than a few errors. There were reportedly around 36 errors that took up three pages to describe and included imagined passages from real cases. When informing the court of the “errors,” the firm said that its policies for using AI were not followed and that the errors occurred even though it had a training course designed to prevent just this sort of thing from happening.

To call this an error is like saying the Titanic running into an iceberg was a slight miscalculation. It’s not just a nanny nanny boo boo.

Why This Is So Bad

So, we can add Sullivan & Cromwell to the ever-growing list of firms that are making these kinds of mistakes and miscalculations. In fact, it’s reported that it’s happened over 1,300 times. And those are just the ones that have been caught. The problem has gone well beyond just all inadvertence and strikes at our fundamental system.

I recently attended an American Bar Association conference. I listened to a panel discussion about the role of lawyers not only in the development of our legal system but as guardians of the rule of law and a fair and impartial system of justice. The panel was composed of Michelle Behnke, the current ABA President, Hon. Bernes Aldana, who was until recently the President of the National Judicial College, and Mario Sullivan, the President of the ABA’s Civil Rights and Social Justice Section.

The panel made the important point that the legal profession has an obligation to teach and educate the public about the rule of law and why it is so important. And why it’s critical that disputes can get resolved fairly and impartially.

Our system of justice hinges on the public’s trust and confidence in our legal system and the bedrock concepts on which it’s based. That’s why we need to take responsibility to make sure that the public understands what these concepts mean and why they are so important.

So, what does this have to do with the Sullivan & Cromwell error? Aldana made a very perceptive point. In order for our system to function, lawyers have to be candid with the courts and judges. And they have to conduct themselves in ways that bolster public confidence in them and our system.

When we cite cases that don’t exist or cite cases for the wrong proposition, it undermines that trust and confidence. When judges can’t have some degree of trust in what both sides tell them, it disrupts their ability to be fair and impartial. Not to mention what happens if you can’t at least have a modicum of trust in what your adversary says.

And when the public hears about lawyers citing cases and laws that don’t exist, they conclude the whole system is a sham. It falls right in line with the inability to trust any video or audio recording and with the shameless lies that permeate social media. It makes it easy to conclude our system of justice is just more of the same. As Aldana put it, “the legitimacy of our courts depends on the public’s trust and confidence.” This is serious stuff. It’s a further erosion of trust in the rule of law. 

I was trying to explain the hallucination problem to someone recently who is not a lawyer and has no legal training. Their first reaction: “Oh, so the lawyer just made up cases to try to win their lawsuit.” Of course, that’s not what happened. But the fact that it was the immediate and initial reaction says a lot. It demonstrates why we need to scrupulously avoid even the appearance of impropriety.

The Devil Made Me Do It

AI can do wondrous things. It can help lawyers work better and faster. But it’s well known (or should be well known by now) that it has a dark side: it makes things up. When accuracy is critical, that’s a real problem.

We can talk a lot about why hallucinations and inaccuracies still show up frequently in court filings. Time pressure, the way that we have historically gotten work done as we have discussed, the time necessary to check citations that often negates the AI advantage in the first place as we have also written. Add to all this the fact that it’s easy and tempting to use AI to get information quickly and it’s easy to see why we have a problem.

But that doesn’t solve the problem and may in fact compound it. Look at it this way: if an associate in a law firm or a judicial clerk pre-GenAI cited a case that didn’t exist, they’d be fired and likely reported to the bar. We seem to think that citing a fictitious case that GenAI provides somehow provides a bit of excuse for what happened. Yes, courts are getting serious about fining lawyers. But the lawyers offer the old “the devil made me do it” excuse.

So, we still see the same thing happen over and over. The public? It’s rolling its eyes at us. Our stature is falling. And by undermining our stature, it undermines what we do, our system of justice, and the rule of law concept on which it’s all based.

When a leading law firm in the world with all its training and policy bells and whistles cites a nonexistent case, it’s a problem for all of us.

What Can We Do?

So, what do we do? Of course it starts with education. Been doing that. We need guidelines and policies. Done that. We need to fine the reprobate lawyers that do this. Doing that.

We need to get stricter. Lawyers that cite hallucinated cases perhaps need to be reported to the bar and disciplined. When a lawyer steals clients funds, the bar comes down hard. Why? It undermines the client relationship and trust. It’s bad for the client and it’s bad for the profession. The same is true here.

We need to mandate tools that enable citation checking quickly and easily, as we have written. It may even mean limiting the use of AI in certain circumstances.

It means doing things like the ABA is doing: focusing on our collective responsibility to protect the integrity of the rule of law. We need to understand that improper reliance on AI isn’t just bad for the lawyers that do it. It’s bad for the profession and our system of justice.

If we don’t grasp the relationship between the rule of law and citing fictitious cases, one day we may wake up in a land where citing fictitious cases won’t matter because there is no rule of law.

---

Stephen Embry is a lawyer, speaker, blogger, and writer. He publishes TechLaw Crossroads, a blog devoted to the examination of the tension between technology, the law, and the practice of law.