Staff Applied Scientist - Dashboards
Staff Applied Scientist defining evaluation strategy and quality metrics for Datadog's AI-native Dashboards product. Owns ML/GenAI evaluation systems, builds datasets and harnesses, and drives improvements in retrieval, tool selection, and agent performance.
About the job
What You’ll Do
- Own the evaluation strategy for Dashboards, as well as sister teams within our organization. Define the metrics — offline and online, quality and cost, single-turn and multi-turn — that the team and the broader organization optimize against.
- Build the eval datasets, golden traces, and regression harnesses that catch quality changes before they hit customers, and make those assets reusable by every team that is building dashboards and widgets through agents.
- Drive measurable improvements to retrieval relevance, tool-selection accuracy, and context efficiency, partnering closely with the engineers on the team.
- Provide technical leadership across the Dashboards team and the broader organization through design reviews, working groups, and mentorship.
Who You Are
- BS/MS/PhD in a scientific field, or equivalent experience.
- 10+ years of relevant engineering or applied science experience, including time as a technical lead.
- Proven track record of leading ML or GenAI initiatives in a product-driven environment, from research through production.
- Significant experience with evaluation, experimentation, or measurement of ML systems at scale.
- Strong product mindset and comfortable driving initiatives across cross-functional teams.
- Thrive in ambiguity and can make sound technical calls when the path isn’t yet defined.
Skills
Machine Learning, Generative AI, Evaluation Frameworks, Experimentation, Retrieval Systems, Tool Selection, Context Efficiency, Regression Testing, Technical Leadership, Cross-Functional Collaboration
Similar jobs
ML Engineering jobs
Square
San Francisco, CA
Build and operate production machine learning systems for ranking, retrieval, recommendations, personalization, and customer intelligence. The role requires 12+ years of production software and ML experience, strong expertise in intelligent systems, and sound judgment around trustworthy customer-impacting signals.
United States
Leads the technical direction and development of large-scale, GenAI-powered recommendation and feed-ranking systems. Requires 10+ years of industry experience in relevance-driven products, deep expertise in machine learning and recommendations, and strong organizational influence and mentoring skills.
United States
Leads technical strategy for Reddit’s Ads ML Platform, improving feature development, training-data generation, experimentation, and the path to production ML serving. The role requires 8+ years in infrastructure or distributed systems, production ML platform experience, and strong cross-team technical leadership.
Garner Health
New York, NY
Leads the reliability, architecture, deployment automation, and monitoring of production machine learning systems. Requires 7+ years of software engineering experience, deep MLOps platform expertise, and strong Kubernetes, cloud, infrastructure-as-code, and observability fundamentals.
United States
Leads the technical direction of large-scale ML infrastructure for embedding, recommendation, and personalization systems. The role requires 8+ years of ML engineering experience, expertise in deep learning and distributed training, and strong leadership across research, infrastructure, and production deployment.