Safeguards Enforcement Analyst, Violence & Extremism

Key Responsibilities

• Design and architect automated enforcement systems and review workflows that scale effectively while maintaining high accuracy.
• Develop and maintain evals that measure model performance on policy areas related to violence, extremism, weapons, and dangerous technology; surface regressions and inform policy and model improvements.
• Partner with Engineering and Data Science to optimize detection and automated enforcement systems for potential policy violations.
• Review flagged content to drive enforcement decisions and surface policy gaps, with particular attention to novel or technically sophisticated misuse attempts, emerging extremist movements, ideologies, and mobilization tactics.
• Support the Safeguards policy design team by providing structured feedback on policy gaps and enforcement ambiguities based on real enforcement scenarios.
• Develop and maintain enforcement guidelines and reviewer documentation that enable accurate, consistent enforcement across a wide range of content.
• Keep up to date with emerging threats, terrorist and extremist movements, regulatory changes, and AI policy enforcement best practices; apply these to inform workflows and evals.
• Identify and escalate emerging misuse patterns, novel attack vectors, and signs of coordinated violent extremist activity.

Minimum Qualifications

• Experience in policy enforcement, threat intelligence, counterterrorism, government, or a closely related field, with direct exposure to harmful content, dangerous technology, violent extremism, or physical harm facilitation.
• Experience standing up and scaling policy enforcement or content review workflows.
• Proficiency in SQL and/or other data analysis tools to draw insights from large datasets and monitor enforcement workflow health.
• Experience identifying emerging risks and threat actors, and communicating findings to diverse stakeholders such as Product, Policy, Engineering, and Legal teams.
• Experience working with generative AI products, including writing effective prompts for content review and enforcement.
• Understanding of the challenges involved in implementing product policies at scale, including in the content moderation space.

Preferred Qualifications

• Subject matter expertise in one or more high-stakes harm areas, such as weapons and dangerous technology, violent extremism, terrorism, autonomous systems, or critical infrastructure protection.
• Familiarity with relevant legal and regulatory frameworks governing dangerous technology, critical infrastructure, or domestic/international terrorism.
• Experience developing evals or red-teaming AI systems, particularly for harmful content or policy enforcement use cases.
• Experience with threat actor profiling and threat intelligence frameworks (e.g., MITRE ATT&CK).
• Experience tracking threat actors, extremist networks, or misuse patterns across surface, deep, and dark web environments.
• Experience with large language models and an understanding of how AI technology could provide meaningful uplift toward serious harm.
• Proficiency in Python for data analysis and workflow automation.
• Background in law enforcement, national security, defense, counterterrorism, or a relevant regulatory environment.
• Experience assessing the technical plausibility and real-world harm potential of content, including distinguishing between general educational content and genuine operational uplift, and between protected speech and genuine incitement/mobilization.
• Familiarity with cross-platform threat analysis and OSINT techniques.

Compensation & Logistics

• Annual Salary: $285,000–$330,000 USD
• Minimum education: Bachelor’s degree or equivalent combination of education, training, and/or experience in a relevant field.
• Location-based hybrid policy: Expect all staff to be in one of our offices at least 25% of the time (some roles may require more).
• We encourage applications even if you do not meet every qualification.