Cyber Evaluations Engineer
Build and operate evaluations for cyber capabilities and safeguard robustness in AI models, analyze adversarial data, and develop cyber-abuse detection probes. The role requires hands-on cybersecurity experience, Python proficiency, evaluation expertise, and strong cross-functional communication.
About the job
Responsibilities
- Design and run capability, uplift, and safety evaluations assessing cyber-relevant risk in new models.
- Execute safeguard-robustness testing before major model launches.
- Analyze evaluation results and communicate findings to cross-functional stakeholders.
- Design, prototype, and tune detection probes for cyber misuse.
- Collaborate with cyber policy partners to translate policy lines into layered abuse-detection architecture and measure precision and coverage over time.
- Build and maintain internal tooling for running and scoring evaluations.
- Work with policy and engineering partners to translate evaluation findings into safeguard improvements.
Requirements
- Experience building or running evaluations, benchmarks, or test suites for software or ML systems, including delivering results on short, fixed timelines.
- Hands-on cybersecurity experience, such as CTF participation, vulnerability research, exploit development, or security research.
- Proficiency in Python.
- Strong communication skills with cross-functional and policy stakeholders.
- Bachelor's degree or equivalent combination of education, training, and experience in a relevant field.
Nice-to-haves
- Deep offensive-security or security-research experience, including building AI security benchmarks.
- Experience analyzing adversarial or abuse data, including jailbreaks, prompt bypasses, intrusion telemetry, or fraud telemetry.
- Experience testing or evaluating systems with government partners onsite.
- Experience with AI/ML evaluation frameworks.
- Familiarity with coordinated vulnerability disclosure practices.
- Experience testing pre-release or pre-deployment software or models under confidentiality constraints.
- Experience authoring detection content such as Sigma, YARA, Suricata, or SIEM rules, or building ML-based abuse detection.
- Active secret security clearance or higher, or eligibility to obtain one.
Compensation
- Annual salary: $300,000–$405,000 USD.
Skills
Python, Cybersecurity, Security Research, Offensive Security, Exploit Development, Ai Security Benchmarks, Machine Learning, Ai/Ml Evaluation Frameworks, Jailbreak Detection, Vulnerability Research, Sigma, Yara, Suricata, SIEM
Similar jobs
Security Engineering jobs
Anthropic
San Francisco, CAConduct offensive security operations, red-team engagements, penetration testing, and adversarial simulations across cloud, endpoint, and bare-metal environments. The role requires at least five years of experience, strong engineering skills, and expertise across multiple security domains.
Anthropic
New York, NYThis hands-on security engineer will research sophisticated threat actors, build intelligence pipelines, conduct threat hunts, analyze malware and infrastructure, and translate findings into durable detections. The role requires at least five years of cyber threat intelligence or related experience, strong Python engineering, and malware and detection-analysis expertise.
Anthropic
San Francisco, CASenior hands-on security engineer responsible for endpoint hardening, device trust, identity and SaaS governance, and scalable corporate-security automation across a growing organization. Requires endpoint security expertise, Python scripting, IAM and zero-trust depth, and strong threat-modeling judgment.
OpenAI
San Francisco, CA
Design and ship security-critical software and firmware at the boundary between policy systems and hardware-backed cryptographic protection. The role requires 5+ years of secure embedded development and deep C, C++, or Rust experience.
Fluidstack
New York, NYThe Security Engineer will track advanced adversaries targeting frontier AI infrastructure, build intelligence pipelines, conduct threat hunts, and create production detections. The role requires hands-on malware and infrastructure analysis, production programming, and close collaboration with detection and incident response teams.