Security Engineer, Anti-Abuse

Responsibilities

• Own the abuse vision for Arena: what gets detected, what gets enforced, how fast, and with what false-positive budget
• Design and operate detection for bots, sybils, coordinated inauthentic voting, and rating-system manipulation
• Build enforcement primitives (rate limits, challenges, shadowbans, account actions, model-side refusals) that are reversible, auditable, and humane
• Detect and mitigate inference abuse and cost exploitation at the platform layer
• Build jailbreak and multi-provider misuse detection across the models Arena serves, and partner with model-provider trust & safety teams on signal-sharing and escalation
• Scope and implement abuse monitoring for every new product launch — web search, web fetch, live site deployment, and whatever's next — as part of the launch checklist
• Prototype and mature into production systems of detection, review, and enforcement for the highest-severity harms (CSAM/NCII, violent extremism, self-harm), including the legal reporting pipeline (e.g., NCMEC)
• Build internal investigator tooling so policy, on-call, and future T&S analysts can triage incidents without engineering bottleneck
• Partner with Security on shared surface — account takeover, credential stuffing, API-key abuse, and the identity/behavioral-signal platform
• Partner with policy, legal, and leadership on acceptable-use policy, enforcement escalations, and public-integrity narrative

Requirements

• 6+ years of production software engineering experience, including building and operating systems under adversarial conditions
• Shipped experience in at least one of: trust & safety, anti-abuse, anti-fraud, anti-spam, integrity, or risk engineering
• Strong SQL and data-analysis skills — this role is 30%+ pattern-finding and investigation, not just shipping code
• Adversarial and investigative mindset — you can articulate a novel attack before designing the defense, and follow evidence when a novel harm surfaces
• High judgment on false-positive cost, user harm, and the reversibility of enforcement actions
• Proficiency in a modern backend language (Node.js, TypeScript, Python, or Go)
• Excellent communication — you'll build alignment with engineering, product, policy, and leadership routinely

Nice-to-Haves

• Experience with LLM-specific adversarial inputs — jailbreaks, direct and indirect prompt injection, tool-use abuse
• Experience with agent safety, browser-automation abuse, or LLM-API abuse
• Background in securing voting, rating, reputation, or marketplace platforms against coordinated manipulation
• ML or ML-systems experience — feature engineering, online/offline evaluation, label acquisition, drift handling
• Experience building investigator or analyst tooling used by non-engineers
• Contributions to open-source trust & safety, abuse-detection, or adversarial-ML work
• Background in gaming integrity, ad-fraud, or financial-crime engineering at scale