# Data Scientist/Engineer – Online Metrics

**Company:** [Perplexity](https://hotfix.jobs/companies/perplexity)
**Location:** New York, NY, San Francisco, CA
**Role:** Data Science
**Experience:** 4+ years
**Skills:** Python, SQL, Spark, Databricks, A/B Testing, Statistical Analysis, BI Tools, LLMs, Data Pipelines
**Posted:** 2026-02-13

> Develops and validates online metrics from user interactions to measure answer quality in LLM-based search products. Builds scalable data pipelines, analyzes A/B experiments, and collaborates with Product and Search teams. Requires 4+ years experience, MS, Python/SQL proficiency.

## Job Description

## Responsibilities
- Discover and validate online signals from user interactions that serve as reliable proxies for true answer quality.
- Design and implement novel online metrics to be tracked both in A/B testing and on product health dashboards, ensuring alignment with ground-truth evaluations.
- Analyze experimental results to validate these metrics, ensuring they accurately predict user satisfaction and drive product decisions.
- Build and maintain the data pipelines that calculate these metrics at scale, delivering actionable quality signals to Search, Product, and model training teams.
- Communicate findings and bring clarity through close collaboration with Product and Search teams.
- Operate in a small, high-impact team where your work directly shapes how Perplexity measures and improves Answer Quality.

## Qualifications
- MS in a technical field or equivalent experience.
- 4+ years of experience working as a Data Scientist, Analytics Engineer, or related role.
- Experience working on search, recommendation, or LLM-based products, with an emphasis on designing online metrics and analyzing A/B experiments.
- Strong proficiency in **Python** and **SQL** (expected to write production-grade code).
- Deep knowledge of statistical analysis.
- Experience with Business Intelligence (BI) tools for visualization and reporting.
- Comfortable with agentic coding workflows and using AI-assisted development tools to iterate faster.

## Preferred Qualifications
- Proficiency with **Apache Spark** and **Databricks**.
- Experience with the development or validation of LLM-as-a-judge systems.
- Prior work supporting customer-facing products at scale.

## Similar roles

- [Quantitative Intelligence Analyst](https://hotfix.jobs/jobs/6cbbee7d-f875-436e-b027-7552bf005b53) - OpenAI - San Francisco, CA - $198k – $320k/yr
- [Data Scientist](https://hotfix.jobs/jobs/2f57246f-d8e8-41b2-9e23-365c11ee3861) - HappyRobot - San Francisco, CA
- [People Research Scientist](https://hotfix.jobs/jobs/46cebc59-4292-4fe0-bf6d-dae345a01aa5) - Anthropic - New York, NY - $285k – $380k/yr
- [Data Scientist, Optimization](https://hotfix.jobs/jobs/33a8c4e0-3a13-4e94-b874-01974295994e) - Lyft - New York, NY - $128k – $160k/yr
- [Data Scientist, GTM](https://hotfix.jobs/jobs/b4908245-1fc8-4461-8c64-9bc98cd938d7) - Anthropic - New York, NY - $285k – $380k/yr

**Apply:** https://hotfix.jobs/jobs/a50c8101-c15b-466b-852f-c1ccf70a21b3
**Canonical:** https://hotfix.jobs/jobs/a50c8101-c15b-466b-852f-c1ccf70a21b3