Data Scientist

What You'll Do

• Define and track product, feature, and agent-level metrics.
• Design, run, and interpret A/B tests for model changes, prompts, agent behavior, workflows, and product features.
• Measure how the performance of our agents affects client outcomes, such as task completion, operational efficiency, response quality, and automation rates.
• Connect offline model evaluations with production performance and real-world customer impact.
• Conduct deep analyses across conversations, workflows, and product usage to identify opportunities and explain differences in performance.
• Investigate anomalies and regressions, perform root-cause analyses, and recommend improvements.
• Build statistical models, simulations, and analytical frameworks to support product and ML decisions.
• Partner with Engineering to improve instrumentation, data quality, experimentation systems, and analytical data models.
• Build dashboards and self-serve tools that help teams understand product and agent performance.
• Communicate findings and recommendations clearly to technical and non-technical stakeholders.

Must Have

• 4+ years of experience in Data Science, Product Analytics, or another highly quantitative product role.
• Strong experience with experimental design, A/B testing, statistics, causal inference, and hypothesis-driven analysis.
• Advanced proficiency in SQL and Python.
• Experience defining and operationalizing product and feature metrics.
• Ability to translate ambiguous product questions into rigorous analyses and actionable recommendations.
• Strong product instincts and the ability to distinguish statistical significance from meaningful product or customer impact.
• Experience partnering closely with Product, Engineering, or Machine Learning teams.
• Strong written and verbal communication skills.
• High attention to detail and commitment to analytical accuracy.
• Founder mindset: ownership, independence, curiosity, and willingness to go deep.

Nice to Have

• Experience working with large language models, AI agents, generative AI, or other probabilistic ML products.
• Experience measuring the production impact of model, prompt, retrieval, or orchestration changes.
• Familiarity with ML evaluation systems and the relationship between offline evaluations and online metrics.
• Experience analyzing conversational, NLP, speech, or other unstructured data.
• Experience with enterprise or B2B products.
• Experience combining quantitative analysis with qualitative methods such as conversation reviews, customer feedback, surveys, or user research.
• Familiarity with modern analytics infrastructure, data warehouses, experimentation platforms, and business intelligence tools.
• Prior experience in a fast-growing startup or other highly ambiguous environment.