Courts crack down on AI hallucinations

Misuse of artificial intelligence in the legal system is escalating from embarrassment to a serious criminal issue, and the problem is expected to get worse before it gets any better.

The country’s highest court has issued two warnings to self-represented individuals over cases featuring artificial intelligence hallucinations this year, threatening possible obstruction of justice or contempt of court proceedings.

Contempt of court can result in a fine of up to $25,000 or six months imprisonment, while the more serious obstruction of justice charge can result in prison terms of up to seven years.

Instances of artificial intelligence hallucinations can be found throughout the legal system, ranging from the Employment Relations Authority and tenancy and immigration tribunals through to the Supreme Court.

These hallucinations typically cite case law that doesn’t exist or misrepresent judgments of little to no relevance.

The Supreme Court fired off its most recent warning earlier this month in a court case between property investor Liyun Chen and finance companies Goodmore Investments and Tawa Trade Finance.

In her self-represented filings, Chen referenced cases that didn’t exist, such as “Peterson v Forbes” or misrepresented the relevance of others, such as the Mainzeal trial.

In dismissing the case, Supreme Court Justices Dame Ellen France, Sir Stephen Kós and Forrest Miller wrote, “Many of Ms Chen’s authorities are irrelevant or do not exist – in either case appearing to be hallucinations of a generative artificial intelligence application.”

It went on to reference a ruling earlier this year relating to a family court case that escalated to the Supreme Court.

“Reliance on false citations, including the unverified outputs of AI applications, may in serious cases amount to obstruction of justice or contempt of court.”

This warning originally came from Jones v Family Court at Whangārei, where the litigant filed submissions that cited sources the court said, “appear to have been hallucinated by an Artificial Intelligence (AI) application“.

“Misuse of AI in legal proceedings has serious implications for the administration of justice and public confidence in the justice system. Persons filing submissions in court must ensure all authorities referred to are genuine and correctly cited,” Justices France, Kós and Miller wrote in a judgment dismissing Jones’s case.

University of Auckland senior lecturer Dr Joshua Yuvaraj says similar warnings have come out of international courts, most notably the High Court of England, so it isn’t surprising to see it happen here.

Mills Lane barrister David Friar says hallucinated case citations and references are becoming increasingly common, “We’re just seeing it more, both in the court system and generally.”

Such hallucinations outside of the court system include a scandal where Deloitte was found to have used artificial intelligence during the creation of a report for the Australian Federal Government.

This resulted in the AU$440,000 report featuring hallucinations, including wrongly referring to a Federal Court case and misquoting the judge involved.

It also got the name of another judge wrong.

Lane Neave special counsel Bethany Frowein has heard anecdotes of hallucinations being filed by reputable law firms, identified and remediated early on in the process.

She reckons the Supreme Court warnings, coupled with better knowledge of artificial intelligence, mean there’s a low risk of it happening again.

But Yuvaraj says given what has happened in other countries, he would be surprised if a lawyer wasn’t penalised or brought up for this sort of thing, “Just due to the sheer weight of numbers, and also the fact that the nature of hallucinations is their ability to produce plausible sounding nonsense.

“The difficulty with the hallucinations is the mistake is not immediately apparent until and unless you closely read it. It’s not going to be standing out at you like a massive typo or an offensive word, it’s going to be much more subtle, but no less damaging.”

But this doesn’t mean lawyers are going to stop using artificial intelligence.

Friar says it can and is a helpful tool for lawyers and even self-represented litigants, “There’s absolutely a place for the use of AI.

“But the key point, whether the output is generated by AI or anything or anyone else, that output and those references to cases and authorities need to be verified before a submission is put towards the court.”

Frowein has found errors during her own use of specialist legal artificial intelligence tools, “I’ve definitely had quite a few cases where, when I’ve manually checked what the AI is telling me, I’ve had to change or update the results because it’s just not accurate yet.

“I’d say it’s creating efficiencies, but definitely way off replacing actual junior lawyers.”

A self-represented litigation, otherwise known as a lay litigant, might not have the ability to check how accurate artificial intelligence is, particularly when it is misrepresenting a genuine case rather than completely inventing one.

“AI continues to get smarter and smarter, and someday in the future it may progress to the point where it is sufficiently reliable to produce legal submissions for lay litigants, but I guess the warning by the Supreme Court is that it’s definitely not there yet, and people still need to use lawyers for their legal research,” Frowein says.

“I don’t think it’s all doom and gloom, I think it’s a tool that’s continuing to improve, and maybe one day it will dramatically improve access to justice, which obviously we need at the moment.

“Litigation is so expensive.”

Friar isn’t at all surprised to see the court issuing warnings, particularly because artificial intelligence “can sound so persuasive and authoritative.”

Contempt of court can depend on the state of knowledge of the individual filing it, so lawyers caught citing hallucinations (as has happened in other jurisdictions) are probably more likely to face consequences.

“Certainly for a lawyer, the court will not look kindly at all on hallucinated cases being cited, and I would expect sanctions to be imposed on a lawyer who does cite hallucinated AI cases,” Friar says.

“For a self-represented litigant, the courts may be thinking about the balance to be drawn in that AI gives self-represented litigants access to tools to participate in the legal system that they otherwise would not have had.”

Yuvaraj’s comments echo this, saying courts internationally have come down hard on lawyers, but self-represented litigants have been met with a more educative approach.

“It’s likely that if this keeps continuing, the courts will lose patience.

“However, the sense is that the use [of AI] would need to be fairly egregious, quite significant, and widespread to sort of warrant some of those more significant penalties.”

The same rules technically apply to everybody who files documents in court, and Yuvaraj says comments being made in judgments and in guidance to lawyers and delegates basically say you stand behind whatever you present to the court 100 percent.

“Now with lawyers, that’s a paramount obligation, really. Lawyers have an utter obligation not to mislead the court, and so that’s why the courts have been more critical and harsher towards lawyers who do this.”

While there is understandably some more lenience, Friar says self-represented litigants can’t be encouraged to use AI with hallucinated case names in there that aren’t being checked or verified.

“That just creates an unsustainable burden on other parties and on the courts to be going through and checking what self-represented litigants are submitting.”

As artificial intelligence enables tasks to be done quicker, it can increase the amount of time spent on verification, “If people who are submitting AI-generated submissions are not checking it, then that burden falls on the court and falls on the other parties to check all of these points and to address them,” Friar says.

“That’s not sustainable for the court system or for other parties that have to be doing all of that checking and response and responding to these sorts of submissions.”

Frowein says that the waste of time extends to the lay litigant trying to rely on hallucinated references, and if any hallucinated material did make it through the system, it could easily be subject to appeal.

The prospect of additional costs being created through AI hallucinations has been raised in a case being handled by the Employment Relations Authority.

After a personal grievance case taken against a Wellington physiotherapy company was declined, the authority said that despite reminders, references made to cases that did not exist were made in the complainant’s final written submissions, resulting in further costs to the respondent.

The complainant was ordered to pay $6500 to the business.

Yuvaraj says courts internationally are beginning to impose wasted costs, “When you receive materials from the other side, and you have to then verify everything, and you find errors and things. Well, that’s wasted time, right?

“So the court is actually awarding costs against the offenders because they’re saying you’re wasting the other party’s time, you’re wasting our time, and you should have to pay for that. So, that seems to be another way they’re trying to deter this conduct.”

Deterrents or not, there’s probably a reason why self-represented litigants are reaching for artificial intelligence to help form their cases.

There’s a couple of key reasons why individuals would represent themselves in court – mainly that they can’t afford a lawyer, or they think it isn’t necessary.

Citizens Advice Bureau legal and strategic national advisor Sacha Green says the cost of taking civil legal action is a real issue.

“There is the challenge that lawyers cost too much and are inaccessible to most people, and so for many people in their pursuit of access to justice, this is a new opportunity for them to try and make use of a tool that might help them.”

“There are problems that come with that, because things are kind of running away on us before we’ve really worked out what all of this means for the justice system and society.”

The Citizens Advice Bureau helped more than 300,000 people last year, with issues raising from tenancies, immigration and legal services.

The organisation largely deals in advice, letting people know their rights and guiding them to legal aid or other forms of community support. A large part of this is intervening and sorting things out before they reach the legal system.

But it is facing budget cuts as councils (their major backers) tighten their belts. Last week the Wellington Branch had its funding cut by 60 percent.

Green says the decision is “hugely problematic” and risks worsening the problems around access to justice in New Zealand.

She doesn’t think artificial intelligence is used with malicious intent, “It might be out of desperation, or it might be out of confusion.”

“I absolutely think for people who are on a track towards going to court, you know, AI is an obvious entice enticing tool for them to use to try and help them navigate what is a really complex system that isn’t really designed for the general public to participate in directly.

“There’s been some efforts to try and support a framework for self-represented litigants, because it should be people’s right, but actually the system isn’t really designed to facilitate that participation.”