Safeguards Enforcement Analyst, User Well-being

Responsibilities

• Support the design and execution of interventions, define key metrics, and curate evaluation datasets.
• Partner with Engineering and Data Science teams to build, tune, and validate detection models for automated intervention systems, including threshold-setting and precision/recall tradeoffs.
• Monitor intervention and detection-system performance over time.
• Review flagged content to drive enforcement and policy improvements.
• Support in-product features that connect users to crisis resources, working with Product, Legal, and external partners on referral pathways and user-facing content.
• Provide detailed feedback to the Safeguards Policy Design team on policy gaps based on real scenarios.
• Track emerging AI policy and external research on AI’s relationship to mental health and use findings to inform decisions and workflows.

Requirements

• Experience in trust and safety, product policy, content moderation, or a related field, with direct exposure to mental health, suicide and self-harm, or related well-being harms.
• Experience designing or running experiments, evaluations, or measurement studies to determine whether an intervention worked.
• Experience translating policy definitions into measurable rubrics, review guidelines, or classification criteria for human reviewers or automated systems.
• Experience managing or coordinating content-review operations, including quality assurance and workflow management.
• Proficiency in SQL and/or other data-analysis tools.
• Experience with generative AI products, including writing effective prompts for content review, classification, or evaluation.
• Experience turning open questions and data into concise, insightful analysis.
• Experience identifying emerging risks and communicating findings to cross-functional stakeholders.
• Understanding of challenges involved in implementing product policies at scale in content moderation.
• Sound judgment in ambiguous, high-consequence cases and comfort making decisions and escalating appropriately when signals are incomplete.

Nice-to-haves

• Subject-matter expertise in mental health through academia, clinical practice, crisis intervention, trust and safety, or related settings.
• Experience building or evaluating LLM-based classification systems.
• Experience using agentic tools, such as Claude Code, to scale analysis or automate recurring work.
• Experience working within crisis support.

Compensation and benefits

• Annual salary: $245,000–$285,000 USD.
• Competitive compensation and benefits.
• Optional equity donation matching.
• Generous vacation and parental leave.
• Flexible working hours.
• Office space for collaboration.
• Minimum education: bachelor’s degree or an equivalent combination of education, training, and/or experience.
• Required field of study: a field relevant to the role, as demonstrated through coursework, training, or professional experience.
• Location-based hybrid policy: staff are currently expected to be in an office at least 25% of the time; some roles may require more.
• Visa sponsorship may be available depending on the role and candidate.