How do you solve a problem like social media? Part II - misinformation

Misinformation and disinformation are among the worst online scourges, with huge real-world impacts. Why are our regulations so weak? And are there risks in making them stronger?

In Part I I looked at social media regulation and the flood of extremism, bigotry and abuse on some of the world’s biggest platforms. If you missed it – or (ahem) didn’t get round to finishing it – you can find it here.

In Part II I’m looking at misinformation and enforcement. Of the harms I’ve discussed across these two posts, tackling misinformation carries some of the greatest risks to freedom of speech. Just because something must be done doesn’t mean anything must.

The cost of inaction - race riots in 2024 (Credit: StreetMic LiveStream)

What you don’t know can’t hurt you

Social media’s role in fomenting extremism is often through circulating misinformation and deliberate disinformation around race, religion, immigration and crime.

Ofcom distinguishes between “misinformation” and “false communications”. The latter is where the communicator knows the information is wrong; the former is where they don’t.

Ofcom and the Online Safety Act (OSA) don’t cover unintentional misinformation at all. But they do regulate false communication via social media, through section 179 of the OSA (excluding Scotland). This sets out that a criminal offence is committed if:

a) a person sends a message,

b) which conveys information the person knows to be false,

c) when sending it, the person intended it to cause “non-trivial” psychological or physical harm to a likely audience, and

d) the person has no reasonable excuse for sending the message

There’s a couple of issues here.

First, unlike the harms I looked at in Part I, false communication is not listed as a “priority offence” in the OSA. Instead it’s a “non-priority offence”. This means that while social media companies are still expected to remove such material when they find it, they don’t have to create risk assessments to consider how their platforms might be used for false communications, or how to mitigate those risks.

This is an odd omission. While acknowledging the flaws in the OSA’s risk assessment system – how it doesn’t force bad faith actors to take action, how hands-off Ofcom is about it – false communication still seems an obvious thing for platforms to risk-assess for. They do have to risk-assess for “foreign interference” – a priority offence – but this only applies when content is posted by or on behalf of a foreign government, rather than social media’s mountains of independently created AI griftslop.

But the bigger issue is around “intent” – whether the perpetrator meant to commit the offending act, and their intention in doing so.

Most criminal offences require some form of intent in order to prosecute. But intent is not as central to the OSA’s hate, threats, abuse and harassment regulations as it is to false communication. For example, platforms alerted to an abusive or threatening post are expected to quickly check it’s not a joke between friends etc, and then remove it. But when confronted with social media content that conveys untrue information, section 179 requires platforms to establish whether the person posting the content knows it is untrue and intends to cause harm with it.

Section 179 is a criminal offence. So far the police have used it sparingly – the need to prove both knowledge and intent beyond reasonable doubt presents a high bar to prosecution. Social media regulation lies outside the criminal justice system, so a lower burden of proof applies, but it remains difficult. As Ofcom’s guidance says: “We anticipate that it will be challenging for service providers to make these judgements based on content alone.”

As I discussed in Part I, social media regulation is underpinned by the principle that what is illegal offline is illegal online, and what is legal offline is legal online. And it would obviously be ludicrously draconian to criminalise people for saying something untrue that they didn’t know was untrue, be that online or offline.

But criminal law and social media regulations shouldn’t necessarily be trying to serve the same purpose. Codes of conduct covering traditional media don’t merely prohibit what is illegal, because they address the media’s role as a source of information. But surely concerns about misinformation on social media similarly relate to social media’s role as a source of information?

Take the case of right-wing internet crank Bernadette Spofforth, who was arrested under section 179 but ultimately not charged after tweeting out a fake name as the identity of the Southport killer two years ago to her tens of thousands of followers, before later deleting and retracting the claim.

Bernadette Spofforth posted, buildings got toasted (Credit: PA Media)

The case was probably dropped because, while it isn’t certain where Spofforth got the wrong name from, there was no evidence she tweeted it knowing it was false, and she insisted she had simply copied and pasted it from another post she believed to be true.

Dropping the charges makes sense in terms of criminal law. But what about online regulation? If moderators at X were alerted to Spofforth’s tweet and confirmed its inaccuracy, it seems nonsensical that the OSA would expect them to delete her tweet if they decided she knew it was inaccurate, but wouldn’t expect them to delete it if they thought she believed it was true. The problem is the misinformation, not the intent.

But if we alternatively required platforms to remove misinformation regardless of intent, we’d soon find ourselves in a different mess. Some of the most controversial and potentially radicalising political issues, such as the ethnicity and immigration status of criminals, or aspects of the Israel-Palestine conflict, are fiercely contested and the subject of disputes over data, context and analysis. In all likelihood, moderators would have little expertise in the subject matter and would head over to an AI chatbot to do their research for them. There’d be little consistency and even less trust over which posts were taken down and which were left up. It would be a disaster.

But given that misinformation is one of the most damaging harms emanating from social media, simply doing nothing because it seems too difficult is also unsatisfactory.

Crisis? What crisis?

Last month, Ofcom published its final proposals for social media regulations in “crisis” situations – defined as an “extraordinary situation in which there is a serious threat to public safety in the United Kingdom” that is likely to cause, or have been caused by, “a significant increase in relevant illegal content and/or content harmful to children on a service”.

It rather conveniently arrived just as criticism of Ofcom was mounting over the misinformation and violence that followed the conviction of Henry Nowak’s murderer, which saw a former police officer forced into hiding after she was falsely accused online of involvement in Nowak’s arrest.

Under the proposed regulations, social media providers must create an “internal protocol” to identify and respond to a crisis, potentially including measures such as increased moderation resources and active monitoring of relevant content. They will also need to open a dedicated communication channel with law enforcement. When the crisis has passed, they must analyse how effective their measures were.

You might think a crisis carrying a “serious threat to public safety” would be the occasion for Ofcom to sharpen its teeth. Apparently not. First, social media providers will ultimately decide for themselves when such a crisis is underway. And nobody will be telling social media providers what content to remove in real time. Instead it will come under Ofcom’s existing, ponderous accountability systems, loosely monitoring compliance via audits and reviews and occasionally investigations. Given X’s testimony to a committee of MPs last year, we can expect Musk’s site to hide behind its Community Notes feature as it always does, regardless of research showing it doesn’t work.

Wifredo Fernandez, X’s former director for government affairs, appearing before a House of Commons committee last year

But at least this crisis mode will tackle misinformation, right? Er… no. The crisis protocols apply to a list of “priority harms” – terrorism, hate, harassment, stalking, threats and abuse, and foreign interference, plus violent content on platforms likely to be accessed by children. Misinformation isn’t specified, and often won’t be covered by “foreign interference”, which as mentioned earlier requires the involvement of a foreign government.

In fact, Ofcom specifically rejected calls to include misinformation within its crisis regulations, as “it would not be appropriate for us to address misinformation or disinformation in these measures where this does not amount to illegal content … as defined by the [Online Safety] Act” (page 16 here).

That leaves us with the section 179 false communications offence, which as we know requires deliberate inaccuracy and intent to cause harm, both of which are difficult to establish.

So two years after social media misinformation in the wake of the Southport murders resulted in mosques being attacked, we have new regulations that barely address misinformation at all.

Targeted strikes

I think we can and should use regulation to target misidentification in a crisis. This is where social media misinformation has the greatest impact, provoking immediate violence and endangering people’s safety. Critically, it also involves scenarios where it is possible to state that something is unequivocally false.

I believe all the following are true – that Ofcom, working with law enforcement bodies, can:

- identify moments of heightened tension around specific incidents

- determine whether certain factual claims are unequivocally true or false

- require social media platforms to quickly remove specific untrue claims about individuals’ identities at times of heightened tensions

In other words, if social media content accuses a named individual of wrongdoing related to an incident causing local or national tension, and the police know it is a case of mistaken identity, platforms ought to be forced to remove those claims. No more former police officers going into hiding.

But even here there are risks. Such powers would almost certainly require sanctions against platforms that refuse to comply. The police could start overusing their new powers, with Ofcom not applying the brakes. While it is in my view acceptable for the state to force the deletion of misidentifications, I think it would be overreach to force the deletion of inconvenient but correct identifications, such as those circulated before the police are ready to make an official statement.

Any powers and associated guidance would need to be explicit about how they can and cannot be used – and pretty much any instance where a forcibly deleted “untrue” claim subsequently turned out to be true would render the powers politically unsustainable.

There are limits to the impact such measures would have. While it may be possible, in the initial aftermath of a murder or terrorist attack, to determine an arrested suspect’s name and potentially their immigration status, religion may be harder to ascertain unless they’re shouting religious slogans, making it harder to assert that claims about a suspect’s religion are unequivocally false.

Also, bear in mind that even if claims the Southport killer was a Muslim asylum seeker had been swiftly deleted, once his true identity and ethnicity emerged we may well have still seen a violent racist backlash, albeit perhaps with different targets.

Henry Nowak (Credit: family handout)

And had the right police officers’ names been circulated in relation to Nowak’s murder, there’d be no misinformation regulations that could apply, and they’d likely have had to go into hiding to avoid vigilante reprisals. Such is Britain in 2026.

The truth is that much of what circulates on social media is just a cruder variant of what’s been pumped out by our national print media for decades. We are living in Paul Dacre’s and Rebekah Brooks’ Britain as much as we’re living in Nigel Farage’s or Elon Musk’s.

AI caramba

Another inviting target for regulation is AI propaganda slop – doctored images, fabricated footage and the like.

This form of misinformation is always deliberate – even if unwittingly shared, it was consciously created at source. And it can be infuriating to see fake photos and videos circulating online of Muslims praying in places they actually weren’t, sparking torrents of vitriolic responses from increasingly rabid Islamophobes.

Some of this content comes from foreign-based accounts churning out provocative AI slop to gain clicks and views that are then financially rewarded by social media platforms. The measures I suggested in Part I, which would increase the liabilities for platforms making these payments, would hopefully lead platforms to terminate these payments, thus killing off a chunk of AI misinformation in one fell swoop.

But other AI content is created for propaganda purposes rather than primarily financial ones.

We should acknowledge that many social media providers have been working to increase the flagging of AI content on their platforms.

What you now see when clicking on a AI video posted on Facebook last year supposedly showing British police bowing to Muslims - the video has 10k likes and 4k shares, presumably accumulated before this notice went up

But perhaps we can go further, using the framework that already exists. Remember that section 179 of the OSA only applies to material posted in the knowledge it is inaccurate, with the intention of causing harm. As I said earlier, criminal law and social media regulations should not be pursuing precisely the same goals as each other.

Therefore, perhaps Ofcom’s false communication provisions might be extended slightly, so that images and footage that have been doctored to change their meaning or generated by AI, with the effect that they look real, and which have the capacity to cause harm, should be taken down by social media platforms unless they are clearly satirical. Even if the person posting the content wasn’t aware it was inaccurate, the person who originally created the content was.

Given that removing online content isn’t the same as pressing criminal charges, we needn’t be overly concerned about the individual culpability of the person posting the content. Instead, we would be recognising that regulating an information environment is different to regulating individual misconduct, and so the “legal offline = legal online” mantra needn’t apply so rigidly on this occasion.

Atlas mugged

There’s another aspect I want to touch on. The government recently announced plans to require social media and video sharing platforms to give more prominence to news content from public service media – essentially the traditional “terrestrial” television channels – “and other trustworthy providers”.

“It could include, for example, a range of national and local news publishers all being more likely to be at the top of people’s social media feeds when they search for news”, the government press release said.

The relevant section of the green paper is worth reading – section 2.2 here. It’s evident the government has an idea of what it wants to achieve but is open to suggestions on how to achieve it – this consultation doesn’t feel like a rubber stamp. “These measures could include news publishers at national and local level, recognising the importance of citizens’ access to a plural range of voices and ensuring we capture local news and voice,” the green paper says.

YouTube has contacted UK content creators warning them the proposals might downgrade their content and encouraging them to respond to the consultation – and, dare I say, kick up a high profile stink online:

What YouTube has been sending to UK content creators regarding the government’s plans to regulate online news

The needs of the creator economy shouldn’t be paramount in the government’s thinking. While light on detail, the government’s aim is in my view correct – to recognise that social media and video sharing platforms are now a major part of Britain’s news ecosystem. Perhaps “early internet” veterans will see this as an infringement of its old anarchistic principles, but those principles have long since been discredited by what is now a billionaires’ anarchy rather than a grassroots one.

It is right that major news platforms should be made to lean towards news sources that have standards of accuracy and systems of accountability. But who gets a look in as a “trusted provider” will be fiercely debated. Clearly we should not have a system where politicians are deciding which individual news providers are trusted and which are not.

The green paper raises the possibility of using the OSA’s existing “recognised news publisher definition” – meaning broadcasters and UK based entities “whose primary purpose is publishing news-related material, created by different persons, subject to editorial control, a standards code, and with a mechanism for resolving complaints”.

I think another option could be to base the new system on holders of an Ofcom broadcast licence, with its requirements of “due impartiality” (currently barely enforced on GB News). “Recognised news publishers” under the OSA don’t have these impartiality requirements. So it would come down to how much the government wants to protect balanced coverage as we enter a social media news ecosystem over the coming years.

We could in theory apply the wider OSA definition to text-based social media platforms, with Ofcom licence holders favoured on video sharing platforms – this would maintain Britain’s historic division between broadcast impartiality and a more diverse and partisan print media. But the green paper specifies “plurality of views” as one of its guiding principles. We shall see.

Get yer trousers on

Finally, there’s the small matter of enforcement – it’s all very well creating new rules, but they’re worth nothing if they get left on the shelf. In particular, there’s the question of how to hold platforms responsible as publishers, as discussed in Part I.

Enforcement should be via Ofcom rather than the courts. That’s partly to avoid adding to the burden on our creaking court system, and partly so multiple failings can be treated together by Ofcom, rather than having the farce of Meta being hauled before the courts over one racist post by Extremely Divorced Dave – or X being taken to court for a new offence every hour.

Ofcom could receive complaints from users that their reports of illegal content had been ignored, and/or it could actively monitor major platforms during periods of heightened illegal activity and report illegal content to the sites. Platforms could be subject to fines periodically – every quarter or every six months, say – for illegal content that had not been removed, factoring in how isolated or repeated such instances were. That would be on top of the existing OSA regulatory regime and enforcement measures.

Not perfect, but better

Fundamentally though, both the existing system and any new regulations need a properly functioning Ofcom rather than the Ofcom we currently have, which drags its feet on tackling social media fraud, and whose recently departed chair saw GB News as the voice of the majority as opposed to being a repeated violator of broadcast media standards.

Baron Grade of Yarmouth, recently departed chair of Ofcom

Incoming Ofcom chair Sir Ian Cheshire has signalled a readiness to square up to “tech bros”, but proper social media regulation will need aligned attitudes at every level of the institution, and – critically – more staffing and resources. More resources means more money, but it’s probably cheaper than policing and prosecuting a new round of race riots every few months.

Above all, Ofcom would need to be ready to use its existing powers to impose substantial penalties on large social media platforms that consistently fail to play by the rules. It has demonstrated a willingness to take on even the big boys when it comes to protecting children. But radicalisation, violence and abuse among adults matters too.

The ultimate power under the OSA is blocking access to a site from the UK, effectively shutting down its UK activity. It is also one with potentially significant political implications. There’s a debate to be had about whether a ministerial veto would make it more or less likely this power is used when necessary – a veto means potential political or diplomatic interference, but without a veto Ofcom may not dare apply the nuclear option to powerful entities at all.

With a reformed Ofcom system, we could potentially see the end of overseas grifters making money through racist AI slop. We could see social media platforms forced to delete incitements to violence. We could have platforms forced to remove misidentification of individuals in times of crisis. And we could see the existing OSA functioning properly to kick into line those social media platforms that think they are above the law, so that the torrents of harassment and abuse that we now instinctively associate with social media are actually clamped down on.

It wouldn’t solve everything – nor should it try to. The problems caused by social media are not just for Ofcom to fix – they engage how we approach loneliness, mental health and community spaces, and they require politicians to argue for pluralist and democratic values against those who oppose them. You can’t just end history through regulation.

None of this would be simple. Much of it would be controversial. But that is what policymaking is meant to be for – finding answers to problems, rather than despairingly declaring them too difficult to solve.