Nov 22, 2024 6 min read

The prevalence of AI-generated sexual content, Meta's new safety model & Carr appointed

The week in content moderation - edition #272

Hello and welcome to Everything in Moderation's Week in Review, your need-to-know news and analysis about platform policy, content moderation and internet regulation. It's written by me, Ben Whitelaw and supported by members like you.

In this week's stories, you should be able to see how the tools and policies meant to keep us safe online are being questioned, reshaped, and, in some cases, outright challenged. Whether it's Snap's research or Brendan Carr's vision, it's clear that the way Trust & Safety teams have done things isn't holding.

A hearty welcome to new subscribers from ActiveFence, RunwayML, Google, Patreon, Technomom and Fujitsu. This is where you’ll get your weekly roundup and, on Monday, you'll also receive T&S Insider from Alice Hunsberger (former Grindr and OkCupid, now PartnerHero). Check out her latest piece on smartphones in schools in today's edition.

That's the intros out of the way; here's everything in moderation from the last week — BW

Today’s edition is in partnership with the Tech Coalition, funding new research on AI generated child sexual abuse

The Tech Coalition is funding three new research projects on the misuse of generative AI for child sexual exploitation and abuse. These projects, announced at an event in Brussels yesterday, will help protect children in a rapidly evolving digital landscape.

Discover the research projects

Policies

New and emerging internet policy and online speech regulation

Donald Trump might might not have a said a lot about the Kids Online Safety Act (KOSA) but the state attorneys general are saying it for him. This week, a coalition of more than thirty AGs urged Congress to pass the much-debated bill before the current session ends, as pressure mounts for platforms to do more to safeguard children. Despite bipartisan support in the Senate, KOSA currently faces opposition in the House due to concerns over potential content censorship and, you guessed it, the extent of enforcement powers being granted to state attorneys general.

The Oversight Board has overturned Meta's removal of three Facebook posts featuring footage from the March 2024 Moscow terrorist attack, directing that the content be reinstated with "Mark as Disturbing" warning screens. The Board noted that, despite violating Meta's terror attack policy, the posts' high public interest value and their role in condemning the attack warranted protection under the newsworthiness allowance, especially in Russia, where media isn’t exactly free and fair.

The mood music: The Russian government has put pressure on social media companies to take down content going back to 2021 and required media outlets to cover the Ukraine war only using official government sources. I’m not sure this will do anything to address Russian attitudes towards the US or its platforms.

Also in this section...

Texas Bill Takes Aim at Online Speech About Abortion Pills (Reason)
Everything you need to know about the WSIS+20 Review (Global Partners Digital)

Products

Features, functionality and technology shaping online speech

The open source model game hotted up this week as Meta introduced Llama Guard 3 Vision, a model that does image reasoning and claims to be optimised to detect harmful multimodal prompts. I haven’t had a chance to read the full paper but it says that it outperforms existing moderation tools like OpenAI and Perspective APIs in adaptability and accuracy.

That will please the Llama users, including, er, academic institutions affiliated with the Chinese government, which were reported to be using Llama earlier this month for military tooling.

Also in this section...

Bluesky says it won’t train AI on your posts (The Verge)
Navigating the complexities of AI content safety: insights from China's Basic Requirements for Generative AI Service Security (Taylor Wessing)

Become an EiM member
Everything in Moderation is your guide to understanding how online speech and content moderation is changing the world.

Between the Friday digest and weekly T&S Insider newsletter, I try to help people like you working in online safety and content moderation stay ahead of threats and risks by keeping you up-to-date about what is happening in the space.

Becoming a member helps me connect you to the ideas and people you need in your work making the web a safer, better place for everyone.

Platforms

Social networks and the application of content guidelines

New research from Snap shows that 1 in 4 young adults have encountered AI-generated sexual content online, with less than half reporting offending posts to platforms or helplines. The findings, which form part of its annual Digital Well-Being Index and is set to be released in full next year, also asked teens how they responded to seeing such content with 54% blocking or deleting it, a similar number speaking to friends and family about it and just 42% reporting it to the platform on which they saw it.

Wider view: As Snap notes, there is a worrying trend for low reporting of harmful content for a variety of reasons (lack of understanding and agency for starters). Reporting is part of what I’m referring to as ‘the internet safety gap’ that vulnerable groups such as women, young adults and LGBTQ+ people face, which the way that they want to address harm and the tools they were being provided with to do so are not aligned. Alice and I wrote about this back in August.

Reporters Without Borders (RSF) has filed criminal charges against X/Twitter in France, alleging the platform's complicity in identity theft and the dissemination of disinformation. This action follows ten (yes, ten) unsuccessful attempts by RSF to have a deceptive video — falsely attributed to the BBC and misusing RSF's identity — removed from the platform. The video received 500k views and ‘Patricia’, the user that posted it, has continued to share hate speech and disinformation since, describing Ukrainians as "Nazis" and "barbarians." The case is unlikely to do much for X/Twitter’s EU investigation under the DSA.

If you tuned into last week’s Ctrl-Alt-Speech, you’ll have heard Mike and I talk about anti-boycott advocate Elon Musk, erm, boycotting Facebook in the aftermath of the Cambridge Analytica scandal. Techdirt has the full details.

Also in this section...

People

Those impacting the future of online safety and moderation

Well, it’s official: Brendan Carr is now set to lead the Federal Communications Commission under Donald Trump’s second term.

Why is this interesting for non US politics watchers? Carr has been outspoken critic of what he calls a “censorship cartel” involving Big Tech giants like Meta, Google, and Apple and wants to repeal Section 230.

As CNN report, he has written letters to various platform CEOs with a warning to “review of your companies’ activities as well as third-party organizations”. And he also helped author the Project 2025 playbook with the aim of tackling perceived anti-conservative biases and expanding free speech protections.

Naturally, he’s been cozying up to Elon Musk, signalling a strong alignment with his deregulatory vision. And like Musk, he’s got his wish to play God on what happens next in the United States.

Posts of note (Bluesky edition)

Handpicked posts that caught my eye this week

“That means you don’t need to hold your breath for an evil company to come up with features to sort out content you don't like; developers can do it, freely.” - Edward Perez, X/Twitter’s former director of product management, on what's different about the new place.
“This is a behind the scenes tooling change that if done right, you will never consciously notice, but that will improve things for everyone. Except the assholes.” Bluesky mod queen rahaeli with the juice on the latest release.
“Okay folks, this morning we launched our Kickstarter campaign for our new *amazing* card game, One Billion Users.” - My co-host Mike Masnick has cyour neighbour/cousin/colleague's next birthday sorted.

ADD ENJOY TODAY'S EDITION