Why police have been told to pause their use of AI in court cases

Simon Thorne, Cardiff Metropolitan University

Police forces in England and Wales have been told to pause some uses of generative artificial intelligence (AI) in criminal justice work, including preparing court statements. The intervention follows concerns that commercial AI systems are being used in situations where accuracy, provenance and accountability are critical.

Alex Murray, head of the Police.AI centre has said forces need to slow down and ensure AI systems are properly assessed before they are used in criminal justice work.

The concern is not simply that AI can make mistakes. Generative AI can produce statements that sound plausible but are false. It’s a problem that becomes much more serious when those statements can influence an investigation or form part of a case presented in court.

This has already happened. West Midlands Police used information generated by Microsoft Copilot while preparing material relating to a proposed ban on Maccabi Tel Aviv supporters attending a football match against Aston Villa in November 2025. Copilot referred to disorder at an earlier match between the two clubs. But the match had never taken place.

The incident is a reminder that AI-generated information can look like credible intelligence even when it is entirely fabricated.

When an AI error becomes evidence

The risks are particularly serious in criminal justice because evidence needs provenance. It should be possible to establish where information came from, how it was obtained and why there is reason to believe it is accurate.

Generative AI complicates this. A large language model does not establish whether something is true before producing it. It generates a response based on patterns in its training and the information it has been given. The result can be convincing without being reliable.

Even asking an AI system to explain how it reached an answer does not necessarily provide an audit trail. The explanation may itself be generated rather than a reliable record of the process that produced the original answer.

That raises difficult questions if AI has been used to prepare material for a criminal case. A defence lawyer may ask questions such as: Was AI used? What information was given to it? What did it produce? Was every claim checked against the original evidence? Was anything added, removed or altered? And is there a complete record of the interaction with the system?

Read more: Investigative interviews are key to solving crimes – should AI be helping police with their inquiries?

These questions are already relevant to real cases. A Derbyshire police detective has been suspended and is under criminal investigation following allegations that AI was used to create evidential material. Several rape convictions are being reviewed after allegations that the detective instructed an AI chatbot to produce case paperwork in ways intended to influence charging decisions. The investigation is ongoing and no charges have been brought.

The case illustrates why the issue is not simply whether an AI system can produce an incorrect sentence. If AI-generated material enters an investigation, it can influence decisions before anyone realises that something has gone wrong.

The obvious safeguard is human verification. Important AI-generated material should be checked by a suitably qualified person against the original evidence and independently verifiable facts.

But this creates an awkward trade-off. If every significant claim produced by AI has to be independently checked, some of the time saved by using the technology is lost.

Read more: Will AI tools make better police officers?

In criminal justice, however, this may be a necessary cost. The purpose of verification is not simply to make AI-generated text more accurate. It’s to ensure that responsibility for the evidence remains with someone who can account for where it came from and why it should be trusted.

That’s an important distinction because an AI-generated document can look authoritative even when nobody has established that its contents are correct. A person who just reads the output and decides that it looks reasonable is not providing the same safeguard as someone who checks it against the underlying evidence.

Efficiency?

There may be good reasons for police forces to explore AI. Case preparation is time-consuming, and technology could potentially reduce administrative work, allowing officers to spend more time on other aspects of policing.

Police.AI is already developing tools intended to prepare and quality-check case files. Pilots are beginning in 2026 with a national rollout planned for 2027. The aim is to reduce workloads, speed up charging decisions and get cases to court more quickly.

Read more: Why AI shouldn’t be used even to decide ‘simple’ court cases

But faster case preparation is of little value if errors undermine evidence later on, which then lead to convictions being reviewed or give defence lawyers legitimate grounds to challenge how material was produced. So, the question isn’t whether AI can be useful in criminal justice. It almost certainly can. The more important issue is how much responsibility can safely be handed to these systems.

In areas where an AI error can affect an investigation, a prosecution or someone’s liberty, human oversight cannot simply mean having a person somewhere in the process. It means ensuring that someone remains responsible for establishing that the information is accurate, traceable and fit to be used.

That may make the adoption of AI slower than some would like. But when the stakes include someone’s freedom, speed cannot be the only measure of whether the technology is working.

This article is republished from The Conversation under a Creative Commons license. Read the original article.