I'm Alice Hunsberger. Trust & Safety Insider is my weekly rundown on the topics, industry trends and workplace strategies that trust and safety professionals need to know about to do their job.

This week, I'm thinking about Bluesky's rapid growth and why scaling up T&S startups is uniquely difficult.

Small EiM programming note: Ben is away this week so there'll be no Week in Review on Friday. It'll be back as usual next week.

Get in touch if you'd like your questions answered or want to share your thoughts on today's edition. Here we go! — Alice

Today's edition of T&S Insider is in partnership with Checkstep, the all-in-one Trust & Safety Platform

Staying on top of all the global regulations can feel like a full-time job for Trust & Safety leaders, right? With new updates and regulatory requirements, it’s easy to get overwhelmed trying to ensure your platform stays compliant.

That’s why we’ve put together a Trust & Safety Regulations Cheat Sheet to make things easier for you. Inside, you’ll find:

• A quick overview of key global regulations, including the DSA and OSA,
• Tips on whether to build your own compliance solutions or find trusted vendors.

Three common myths about T&S at startups

I’ve spent my entire career in Trust & Safety working with small-to-medium platforms that are scaling quickly. There are unique challenges that come with T&S at a smaller scale, some of which we’ve seen illustrated perfectly this week with the rapid rise in users at Bluesky after X was blocked in Brazil (EiM #262). One of these challenges is the unrealistic expectations and myths that prevail about how "one easy fix" will solve online abuse once and for all. Here, I discuss three of these myths and why they're wrong.

Myth one: Communities can be cozy forever

Online communities almost always start out nice and cozy. With a small, niche group, there are shared values and social pressure that ensures people tend to get along with each other. But, it's easy to fall into the trap of believing it can be that way forever.

Sometimes there is one unique platform feature which people believe has “fixed” the problem of toxic social media. I remember a dating app founder naively saying that they solved the problem of bad actors by requiring users to link their LinkedIn profile during signup. If only it were that easy!

Recently, we’ve seen Front Porch Forum touted as the solution because they delay posts for a few hours and require people to use their street name. In reality, I think much of the reason why FPF works so well is because their users are Vermonters — sharing a unique (and fairly homogenous) culture that prioritises being reasonable, pragmatic, and taking care of your neighbours.

New platforms that fall into this trap soon learn that scale means they are more attractive playground for trolls, bad actors, and criminals. The truth is that small platforms just aren’t as attractive to these folks (or aren’t on their radar yet). More users bring more problems — it’s often as simple as that.

Just like founders may be lulled into a false sense of security with a cozy community, unfortunately the users are too. People learn to have unreasonable expectations for platforms when they get used to things being utopian for a while, and they have a hard time understanding why things can’t be that nice forever.

Myth two: Automation fixes issues at scale

The cozy phase ends when scale begins. Startups can’t always predict when the scale will come, or what form it will come in, but it's the goal of every startup to reach this phase. It's easy to think that automation can step in and solve issues at scale, but the rapid scale of startups actually makes automation harder. As Aaron Rodericks, head of T&S at Bluesky posted:

Our automations were designed for activity levels prior to the 2.5 million new users. Rules that previously worked now generated hundreds of false positives as a huge proportion of the network is suddenly new accounts under a week old.

The influx has likely shifted Bluesky's demographic composition. While we don't collect specific data, conversations suggest a younger crowd joining. This brings new challenges, like increased discussions around eating disorders.

Not only are the automated rules failing to scale, but the human team takes time to scale as well. In just a couple of days, Bluesky had to revise all of their automation rules and pivot to be able to support Brazilian Portuguese at a scale previously unthinkable to them. Interviewing, hiring, and onboarding moderators takes time (especially if you’re doing it in house, as Bluesky does).

In a situation like this, it's hard enough to react in time and prevent the worst of the worst from flooding a platform, let alone being able to be proactive.

Myth three: no one cares / they're bad at their jobs

When communities go from feeling cozy and safe to being overrun by spam, trolls, and CSAM, it is easy to point a finger at T&S teams and say that someone is asleep at the wheel or doesn’t care. This couldn't be further from the truth. As Bluesky user Rahaeli (former head of T&S at LiveJournal) posted,

People who don't care don't last. You need to have a reason to do this work, or else you won't ever psychologically survive it. It is infuriating when people insist that because someone doesn't solve an unsolvable problem, perfectly and immediately, that they're somehow The Enemy.

Determined bad actors are using every trick in the book to post and find truly horrific content. Some of these bad actors are literally organised crime gangs with more resources than T&S teams at startups. Some are individual trolls or degenerates who are hellbent on using a platform to further whatever agenda they have.

To prevent the worst of the worst with limited resources, startup T&S teams need to take pretty drastic measures with blunt instruments, which unfortunately can mean collateral damage along the way. Examples of this would be fixed rules based on keywords, account signup date, post velocity, etc. When you have more users, though, you also have more legitimate people who are weird edge cases — using the platform exactly as intended, but just in a really atypical way (and often these are power users who will definitely be really vocal and annoying if they’re banned by one of these systems).

On the flipside, just like there are false positive bans, there are also cases where real instances of harm aren’t caught and removed right away. Normal users don’t have any sense of the vast scale of horrific content that moderation teams silently remove every day. Even if a platform catches the vast majority of harmful content proactively, if a small sliver gets through it will be noticed.

This means that people are seeing both ends of the “long tail” of moderation — very vocal “I was banned for no reason!” posters, as well as stuff that seems legitimately bad that hasn’t been taken down. Not to mention the people who get banned for very real reasons but claim ignorance, or people who evade bans and make users think platforms never banned in the first place, or content that some people think should be taken down but isn’t against the ToS.

When compared to the cozy times of the recent past, it can be very easy (and maybe even reasonable) to see this kind of content and assume that no-one cares or that they are bad at their jobs. However, that perspective doesn't reflect the reality behind the scenes. I really commend the team at Bluesky for being as transparent as they are (and for releasing new user-facing tools during this busy stage), and I hope that users understand how hard this work can be.

Also worth reading

The Russia-Funded Youtube Story Will Only Get Bigger (Big Technology Substack)
Also see: We need to know who is funding the creator economy (Taylor Lorenz Substack)
Why? It turns out that some US conservative influencers were being paid by Russia to discuss Russian talking points and influence the election. YouTube has removed the accounts, but there are likely many more that haven't been found yet.

The Impact of Content Moderation on Marginalized Communities (Trustlab)
Why? Lots of good stuff in here including: "While the intent behind moderation is often to protect users from harm, the execution can unintentionally silence the voices of those who are already underrepresented. Understanding how moderation policies affect different groups is crucial for creating more equitable online spaces."

Meta Oversight Board okays calls for violence against Venezuelan ‘colectivos’ (The Verge)
Why? This is a great example of why T&S is so hard. It seems obvious to create policies against calling for violence and death, but it can be more nuanced than people think, especially when there are power dynamics and intense political situations involved.

Building a robust Trust & Safety framework (Tremau)
Why? I love a good framework, and this is a good piece that outlines how startups should think about supporting T&S holistically.