Experts urge Ofcom to probe AI’s role in fake news after major incidents

Artificial intelligence (AI) software used to propagate fake news following significant events, often to generate income for social media users, should be scrutinised in an upcoming investigation into fraudulent advertising, experts have urged.

Researchers at the Alan Turing Institute’s Centre for Emerging Technology and Security discovered that AI played a role in driving some of the fake news disseminated online after the Southport murders, primarily for financial gain. They have recommended that Ofcom, the communications regulator, address this issue during its consultation on fraudulent advertising, scheduled for this summer.

A report published on Wednesday revealed that Channel3Now, a website that initially published a false name for the suspect, was established using a service provider that "markets itself as using AI to generate content for users seeking passive income". The report also found that AI was employed to repackage articles, making them appear more credible.

"This evidence suggests that AI-generated misinformation, with minimal human editorial oversight and monetised through digital ad networks, played a role in injecting divisive falsehoods into the public discourse following the Southport murders," the report stated.

open image in gallery

"AI tools that generate content based on trending topics, optimised for sensationalism and virality rather than factual accuracy, could therefore have an outsized impact relative to the effort required. This suggests that much more focus is needed on undercutting the financial incentives behind advertising networks, which may inadvertently encourage the spread of harmful content."

Current research indicates there are approximately 2,089 AI-generated "news" sites operating across 16 languages, many with "little to no human oversight".

The report further recommends that AI chatbots should automatically flag their fact-checking limitations, particularly in the aftermath of major incidents. It cited instances such as Grok incorrectly labelling a Metropolitan Police video of a Unite the Kingdom protest as fake, a false claim that garnered two million views. Grok also erroneously identified a deepfake image of the Bondi Beach shooting as authentic.

open image in gallery

To mitigate such errors, the report suggests chatbots should display a pop-up warning users that results cannot be relied upon while an incident is still unfolding. It also calls on the Government to establish a crisis response plan for events where an AI "information threat" emerges, and to issue fact-checking guidance to schools, universities, and the wider public via social media.

Sam Stockwell, a senior research associate at the Alan Turing Institute’s Centre for Emerging Technology and Security, commented: "Crisis events are unpredictable and volatile scenarios. Combined with a poorly understood AI threat landscape, this means that we are not currently equipped to deal with this growing threat to public safety. Yet while we need to address the critical risks associated with AI tools in this context, we must also recognise that the same technology can help to strengthen democratic resilience in times of crisis. Actioning the recommendations outlined in this report will go a long way in demonstrating that the UK can protect the public against these threats in the event of future AI-driven incidents."