Skip to content
PerplexityPerplexityNew York, NY

Data Scientist/Engineer – Online Metrics

Develops and validates online metrics from user interactions to measure answer quality in LLM-based search products. Builds scalable data pipelines, analyzes A/B experiments, and collaborates with Product and Search teams. Requires 4+ years experience, MS, Python/SQL proficiency.

Salary not listed
Hybrid4+ YOEData Science

About the role

Responsibilities

  • Discover and validate online signals from user interactions that serve as reliable proxies for true answer quality.
  • Design and implement novel online metrics to be tracked both in A/B testing and on product health dashboards, ensuring alignment with ground-truth evaluations.
  • Analyze experimental results to validate these metrics, ensuring they accurately predict user satisfaction and drive product decisions.
  • Build and maintain the data pipelines that calculate these metrics at scale, delivering actionable quality signals to Search, Product, and model training teams.
  • Communicate findings and bring clarity through close collaboration with Product and Search teams.
  • Operate in a small, high-impact team where your work directly shapes how Perplexity measures and improves Answer Quality.

Qualifications

  • MS in a technical field or equivalent experience.
  • 4+ years of experience working as a Data Scientist, Analytics Engineer, or related role.
  • Experience working on search, recommendation, or LLM-based products, with an emphasis on designing online metrics and analyzing A/B experiments.
  • Strong proficiency in Python and SQL (expected to write production-grade code).
  • Deep knowledge of statistical analysis.
  • Experience with Business Intelligence (BI) tools for visualization and reporting.
  • Comfortable with agentic coding workflows and using AI-assisted development tools to iterate faster.

Preferred Qualifications

  • Proficiency with Apache Spark and Databricks.
  • Experience with the development or validation of LLM-as-a-judge systems.
  • Prior work supporting customer-facing products at scale.

Skills

PythonSQLSparkDatabricksA/B TestingStatistical AnalysisBI ToolsLLMsData Pipelines

Similar roles

Data Science jobs
OpenAI

Quantitative Intelligence Analyst

OpenAISan Francisco, CA

Quantitative Intelligence Analyst discovering novel risks in human-AI systems using quantitative models, data mining, and statistical analysis to develop early-warning signals for trust and safety threats. Requires 3+ years in quantitative intelligence, risk research or trust & safety, plus Python/SQL skills.

198k – 320k/yr
Hybrid3+ YOEData Science
HappyRobot

Data Scientist

HappyRobotSan Francisco, CA

Help build and improve HappyRobot’s AI agent products by defining metrics, designing and interpreting A/B tests and experiments, analyzing conversations/workflows, and turning data into insights that guide Product and ML roadmaps. Requires 4+ years in data science or product analytics, strong SQL/Python, stats, and experimental design experience.

Salary not listed
Hybrid4+ YOEData Science
Anthropic

People Research Scientist

AnthropicNew York, NY +2

Lead rigorous research on employee experience, manager effectiveness, and organizational health at Anthropic. Design studies, analyze surveys with psychometrics, build frameworks, and translate findings into people strategy using SQL, Python/R, and advanced methods. Requires advanced degree and 5+ years quantitative research experience.

285k – 380k/yr
Hybrid5+ YOEData Science
Lyft

Data Scientist, Optimization

LyftNew York, NY

Develop mathematical optimization and ML models for Lyft's Planned Pricing systems to set efficient price targets. Requires PhD in quantitative field, strong optimization experience, and Python proficiency in production environments.

128k – 160k/yr
HybridData Science
Anthropic

Data Scientist, GTM

AnthropicNew York, NY +1

Drive data-informed decisions across Anthropic's commercial customer lifecycle by owning GTM metrics, causal inference experiments, statistical modeling, and actionable insights for acquisition, activation, expansion, and retention of a consumption-based AI platform.

285k – 380k/yr
Hybrid5+ YOEData Science