Data Scientist, Preparedness
Data Scientist evaluates and improves AI mitigation systems, focusing on classifiers and detection pipelines for risks like biosecurity and cybersecurity. Builds monitoring frameworks and drives cross-functional impact using Python, SQL, and analytical expertise.
About the job
What You’ll Do
- Evaluate and improve mitigation systems, including classifiers and detection pipelines across domains (e.g., biosecurity, cybersecurity, emerging risk areas).
- Diagnose false positives and false negatives with deep error analysis, root cause investigation, and clear recommendations.
- Build monitoring and measurement frameworks to track mitigation effectiveness over time and across user segments/use cases.
- Identify trends in over-blocking vs. under-blocking, quantify customer impact, and propose prioritized interventions.
- Develop insights from customer feedback, complaints, and usage patterns to detect shifts in adversarial behavior and system failure modes.
- Expand risk monitoring into new areas, including cybersecurity threats and model loss-of-control/sabotage scenarios, partnering with domain experts.
- Communicate results to technical/executive stakeholders with crisp narratives, decision-ready metrics, and clear tradeoffs.
Qualifications
- Significant experience in data science or applied analytics in high-stakes domains (e.g., security, trust & safety, abuse prevention, fraud, platform integrity, reliability).
- Strong foundations in experimentation, causal thinking, observational inference; design robust measurement under imperfect data.
- Fluency in SQL and Python (or equivalent) for analysis, modeling, monitoring workflows.
- Experience building metrics, dashboards, operational monitoring that changes outcomes.
- Track record driving cross-functional impact with engineering, product, research partners.
Nice-to-haves:
- Cybersecurity data science experience (threat modeling, adversarial dynamics, abuse patterns, security telemetry).
- Classifier evaluation, calibration, thresholding, error analysis at scale; familiarity with detection systems in adversarial settings.
- Trust & Safety experience.
- Interest in AI safety, alignment, catastrophic risk prevention.
Skills
Python, SQL, Data Science, Machine Learning, Classifiers, Experimentation, Causal Inference, Dashboards, Monitoring, Cybersecurity
Similar jobs
Data Science jobs
OpenAI
New York, NYThis senior data scientist will define measurement frameworks for AI-agent security, evaluate controls and security findings, and improve detection and response outcomes. The role requires 5+ years of quantitative experience, strong SQL and Python skills, and experience with cybersecurity or other adversarial-risk domains.
OpenAI
San Francisco, CAThe Data Scientist will shape analytics strategy and deliver forecasting, experimentation, optimization, dashboards, models, and decision tools for Real Estate & Workplace operations. The role requires strong applied statistics, causal inference, SQL, Python, stakeholder communication, and comfort working with ambiguous operational data.
OpenAI
San Francisco, CAPeople Data Scientist focused on AI fairness and bias testing for People systems. Designs algorithmic audits, validation studies, and fairness infrastructure across the employee lifecycle. Requires deep expertise in fairness metrics, statistical modeling, and Python/R/SQL.
Watershed
San Francisco, CA
Develops scientifically rigorous sustainability methodologies that become product capabilities for corporate climate and ESG data. The role combines climate expertise, GHG accounting, standards interpretation, data reasoning, and hands-on collaboration with engineers and product teams.
Plaid
San Francisco, CASenior Data Scientist supporting Plaid’s Credit product area, translating ambiguous product questions into analytics, metrics, experiments, and strategic insights. The role partners closely with product and engineering teams and requires 5+ years of data science or related analytics experience.