I'm Alice Hunsberger. Trust & Safety Insider is my weekly rundown on the topics, industry trends and workplace strategies that trust and safety professionals need to know about to do their job.

I've been writing this newsletter for almost a year, and have really appreciated you all going on this journey with me! Thanks so much to everyone who has sent suggestions and comments along the way.

I want to make sure that I write about the things that matter to you. So Ben and I have teamed up with Katie Harbath (from Anchor Change) to launch a year-end survey for T&S newsletter readers, and we'd really appreciate it if you took five minutes to fill it out.

There are two more opportunities to join our EiM hangout/barn dance, which take place on Fridays at 7am PST / 10am EST / 3pm GMY / 11pm SGT – email me at hi@alicelinks.com if you want to join us. The last one will be on the 20th December and Ben and I will decide in the new year whether we want to start them up again. (What do you think? Let me know).

This week in the newsletter, I'm thinking about over-enforcement and how it's actually a distinct set of problems that require different approaches.

As always, get in touch if you'd like your questions answered or just want to share your feedback. Here we go! — Alice

One person's "error" is another's "censorship"

Meta’s Nick Clegg came out last week to say that Meta’s error rates were too high during the US election, and that “too often, harmless content gets taken down, or restricted, and too many people get penalized unfairly".

His comments come after extensive pressure from right-wing politicians about “content censorship” which led to Mark Zuckerberg writing a letter to Jim Jordan, Chairman of the House Judiciary Committee, back in August (EiM #261). As Ben and Mike discussed in this week’s Ctrl-Alt-Speech podcast, Clegg’s comment should be seen as a political signal to the incoming administration.

However, I’m much more interested in what Clegg means when he says “error rates”.

"Error rates" for a platform can feel like "censorship" to a user. When people perceive their content is being taken down unfairly, they start to lose trust in the platform and its policies, especially if they see other content remain up that they think should be removed.

This is exactly what happened recently when Threads and Instagram were the subject of odd and innocuous "moderation failures" due to a broken moderator tool. It led to wild assumptions and conspiracy theories in which people assumed malice from Trust & Safety departments who often doing the best they can with limited resources.

When “censorship” criticism is levelled at a platform, it usually boils down to one of five reasons. Some of them are more justified than others but it would be helpful for more people — platforms as well as the public— to be specific about which so-called “censorship” they mean.

1. Enforcement errors

Sometimes, the policies are clear, but errors still happen. This seems to be what happened in the Threads/Instagram example I mentioned.

Some of these are rounding-error mistakes because a platform has millions (or billions) of people. Even a 99% accurate system may have millions of mistakes a year. That’s part of the reason why the Digital Services Act and other regulation puts an onus on Very Large Online Platforms (VLOPs).

Other errors could be more systemic, such as bias, lack of training or lack of resources for human moderation teams. They could also be due to machine learning or AI systems not having accurate (or updated) data sets, which is especially an issue during breaking or emerging news events.

Finally, some errors may be due to a platform deciding to index towards recall rather than precision. When it’s critical to catch every single piece of bad content, then some non-violative content will be caught in the net.

A basic example of this is automation through keywords; let’s say a platform wants to make sure a certain slur never appears. They could set a keyword filter to find every single instance of that word and block it, but it will also catch instances of people reclaiming that slur as their own, or discussing why the slur is bad. 

2. Unclear policies or lack of user education

Often, users just don’t understand policies on platforms and don’t know that they’re breaking the rules. Meta shares a good amount of detail in their public-facing policies, but not all platforms are as transparent. Tinder, for example, says they don’t allow sexual content, but doesn’t describe what that means exactly. Apple’s policies that allow an app to be allowed in their app store say they don’t allow “Inflammatory religious commentary” but don’t explain what is defined as inflammatory, or where they draw the line.

3. Differences in opinion over policy

Increased communication and transparency around policies can help when users are confused, but there are also genuine disagreements about the definitions of existing policies. For example, definitions of art and sexual content, what counts as hate speech, or which foreign policies are shaping definitions of terrorism vs. state sanctioned violence. 

4. Gaps in policy

Platforms can sometimes lag behind current events, and need time to update their policies during emerging situations. We clearly saw this during Covid-19 – platforms had a hard time keeping up with emerging information, new conspiracies, and a variety of recommendations from around the world.

During emerging events, moderators must rely on platform guidance, but if that guidance doesn’t exist yet, then enforcement can be inconsistent or delayed. This is especially the case for smaller platforms, and for policies around regions that have less representation at the platform, or for niche communities or subcultures.

5. Third party and distribution-layer policies

Many people don’t realise that the Apple and Google app stores set policy for all the apps they distribute. For example, their fairly conservative stances on nudity (even for 18+ apps) mean that every app needs to have conservative nudity policies as well if they want to be in the app stores.Other distribution-layer services have content policies, such as Cloudflare and AWS, though these are generally limited to blocking illegal content only.

Advertisers also shape content guidelines on platforms through brand safety guidelines, though there is some recent pushback on this from Elon Musk.

So, when a user disagrees with a platform’s policy and wants it changed, it could be that they actually should be pressuring a third party rather than the platform itself.

Avoiding accusations of "censorship"

If "error rates" risks confusing users or feeling like something nefarious has happened, how we can platforms make it clearer what's going on under the hood?

I’ve always thought that there is enormous opportunity for platforms to embrace transparency reports and write them in a way that will educate users and be interesting.  These reports could include more information about why content removal decisions are reversed and what they’re doing to mitigate errors in the future.

However, as reporting become more compliance-based to follow regulatory requirements, platforms are more likely to stick to the letter of the law and not include any additional information that isn’t explicitly asked for. Platforms face risks by being transparent (which, perhaps, is why they don’t take this route), but wouldn’t it be fantastic if there was public communication from T&S departments that felt relatable and human?

At the same time, platform users must be willing to engage in nuanced policy discussions, putting pressure in the right places, rather than resorting to accusations of "censorship" or bias. Right now, it doesn’t feel like we're at the point where there's a willingness to understand or engage in the trade-offs, meaning T&S teams are often scapegoated rather than given more resources to do better. But only through a shared understanding of the realities and constraints of content moderation can we move towards more effective and equitable solutions.

Also worth reading

Moderators Across Social Media Struggle to Contain Celebrations of UnitedHealthcare CEO’s Assassination (404 Media)
Why? An interesting look at how community moderators are making content moderation decisions.

The banks warned her it could be a con. The scammer’s influence was stronger. (Washington Post)
Why? A heartbreaking look at the intense hold that scammers can have over their victims.

Closing the AI equity gap: Trust and safety for sustainable development (World Economic Forum)
Why? I love this take on using T&S as a tool for sustainable development of emerging technology.