Making the best of a T&S incident (part two)

I'm Alice Hunsberger. Trust & Safety Insider is my weekly rundown on the topics, industry trends and workplace strategies that Trust & Safety professionals need to know about to do their job.

Hello from London! I'm here for the Trust & Safety Summit and Marked as Urgent. I grew up not far from here, in Oxford, but haven't been back to England in many years, so everything is making me feel nostalgic. Ben was kind enough to give me a walking tour of London yesterday and, naturally, we took a selfie in front of the most quintessentially English thing we could find.

This week, I'm revisiting how to get buy-in for Trust & Safety, focusing on how to make the most out of a crisis to earn respect and support from the C-Suite when it really matters.

Get in touch if you'd like your questions answered or just want to share your feedback. Here we go! — Alice

Today’s edition is in partnership with Safer by Thorn, a purpose-built CSAM and CSE solution

Powered by trusted data and Thorn’s issue expertise, Safer helps trust and safety teams proactively detect CSAM and child sexual exploitation conversations.

Safeguard your platform and users with proprietary hashing and matching for verified CSAM and a classifier for finding possible novel CSAM. Plus, Safer’s new text classifier provides much needed signals to help trust and safety teams find conversations that violate your child safety policies, such as those containing sextortion or requests for self-generated content from a minor.

Don't put a good crisis to waste

In Part One, I talked about getting buy-in from exec teams for Trust & Safety before an emergency. Unfortunately, that’s not how things mostly get done.

T&S investment decisions more frequently happen in times of crisis and when there’s an exponential risk to the brand/reputation (usually because of bad press). At that point, someone important usually notices — even after T&S folks have been ringing alarm bells for a while — and suddenly resources are made available.

There are a number of problems with this approach: 

• The risks that are raised by these emergencies aren’t always aligned with the actual risks/priorities that T&S teams have identified. Sometimes there are competing risks that undo each other: see my triangle of safety, privacy, and self expression.
• T&S teams often don’t have enough resources to fully complete projects before being swung over to put out another emergency. This can mean jobs are half-done and data is sloppy.
• These reactive projects often aren’t strategic enough to really make a difference to the bottom line in some way, and so leadership underestimates the impact of T&S. When T&S leaders aren’t seen as strategic partners within the business, they can be taken less seriously by management.

However, the reality is that T&S crises are the typical way that major strategic and investment decisions are made: So, with that in mind, here’s my guide for making the most of a T&S incident: 

Before the incident happens

Have an incident response plan that sets out the who, what, when, and how of an incident. This security breach template from Github can be easily modified for a Trust & Safety team. You can start with a basic plan and build from there.

The key things to have in place are detection and triage protocols. This includes:

• Monitoring for incidents (automated alerts, user reports, proactive sweeps, etc). Be as proactive as possible– this is how you find out about things before they become a true crisis. 
• Creating a RACI chart so that everyone knows who to involve during an incident. (And create backup plans for when people are out of the office). 
• Developing a scoring mechanism for categorising incidents by severity and urgency and responding according to that matrix.

Companies are also required to do risk assessments as part of regulatory compliance, which can also help to get you ahead of a crisis.

There are many resources out there on this already, such as the guide from Digital Trust & Safety Partnership, and this from Ofcom. Engaging meaningfully with this process, rather than approaching them as a check-the-box exercise, can be very helpful for prioritisation and resourcing in advance of a crisis. 

During the incident

For anyone who’s been in the midst of a crisis, you’ll know everything is a blur, even if you gave an incident plan. Not everything goes to plan but I’ve found that focusing on the following two things, the more manageable it is:

Investigation

• Use your RACI chart to collaborate with cross-functional teams (customer support, security, engineering, legal, PR, etc.). Work with them to get all the information you can. 
• Identify the root cause of the issue (I love this template from Del Harvey).

Containment/ Communication

• Take immediate steps to limit harm as much as you can (eg. content takedown, account suspension). Of course, this should be proportionate to the actual issue and following your usual enforcement guidelines. I’ve been in situations where an internal stakeholder or VIP user is urgently calling for a specific enforcement action that isn’t appropriate, so it’s critical to slow down at this point and ensure that guidelines are being followed fairly.
• Create a communications plan for each of the key audiences:
  • With internal stakeholders on your RACI matrix (see below as well). Let them know what actions you’re taking, and why. 
  • With affected users (be honest. Give context, ie. scope, severity, urgency, etc and refer to your public documentation showing enforcement guidelines and usual practices). In some cases it may be needed to message a larger group of users; if this is the case then be sure to consult legal and comms teams before sharing anything.
  • With law enforcement, if needed. 

After the incident is over

In order to evaluate your T&S team's response to the crisis, it’s vital to review what happened and what could’ve gone better. Don’t skip over any of the following steps, even if it feels tempting to do so:

• Conduct a cross-functional post-mortem: Was the plan followed? Who on the team needed more training? Did your backup plans work? What areas took a long time to investigate? How well did teams coordinate with each other?
• Document the incident as thoroughly as you can: Don’t write reports that no one will read – what insights will be actionable for other teams? What are they interested in? This is also a great time to include cautionary tales from other companies to show what alternate-universe crisis was averted. 
• Create a plan for long-term fixes (policy changes, tooling improvements, more staff, etc.): You might not be able to get immediate resourcing, but it’s important to have your requests documented. 

For a long time, I never talked about the really stressful, difficult incidents that my teams dealt with. I thought it was my role to shield other people — and senior leaders — from knowing the reality of the work we did, because it can be really upsetting.

However, I realised that this actually didn’t do any good for anyone. I hear from a lot of T&S leaders that they feel as though they’re taken for granted or that no one really understands what they do, and it could be because they made the same mistake I did. 

The best time to show people why T&S exists at your company is after a crisis. If your team really shines during the most intense situations, everyone at the company should know about that and celebrate them. And if your team does badly because they didn’t have the resources they needed, that’s important for people to know as well.

It also helps people in other departments have a more realistic sense of the downsides of the technology they work on. This is one way to start building a culture of safety by design, where engineers and product managers think about the risks in advance, and everyone works towards creating a safer platform for users.

Also worth reading

Platform Regulation Syllabus (Daphne Keller, Stanford Law School)
Why? People often ask me about continuing education for T&S professionals. There's not much out there, but looking at the syllabus covering T&S for law students is a great way to get a thorough overview of topics from a different perspective.

Democrats Team Up w MAGA to Censor The Internet (User Mag)
Why? "As early as next week, Senators plan to introduce the first bipartisan bill to repeal Section 230, the landmark internet law that protects free speech online." Taylor Lorenz does a great job of reporting on T&S issues for a general audience and this is no exception.

The ROI on T&S: What we know (and everything we don’t know) (Rachel Kowert)
Why? An excellent argument for why Trust & Safety is worth investing in for gaming companies.

ROOST reminds us why open source tools matter (Fatima Faisal Khan, Tech Policy Press)
Why? "Through open source, platforms can pool their expertise and rapidly refine tools, benefiting from a global developer community that can spot weaknesses and drive innovation."

AI Slop Is a Brute Force Attack on the Algorithms That Control Reality (404 Media)
Why? "The best way to think of the slop and spam that generative AI enables is as a brute force attack on the algorithms that control the internet and which govern how a large segment of the public interprets the nature of reality. It is not just that people making AI slop are spamming the internet, it’s that the intended “audience” of AI slop is social media and search algorithms, not human beings."

Once You Slop, You Can’t Stop - Ctrl-Alt-Speech

In this week’s round-up of the latest news in online speech, content moderation and internet regulation, Mike and Ben cover:Twitter Inc. Official ‘Bird Logo’ Fascia Sign - An Iconic Fixture from the Company’s Market Square Headquarters in San Fran…

Buzzsprout