Director, Research
Lead AI evaluation for Figma's AI-powered products. Define quality metrics, build human + automated eval frameworks (rubrics, golden datasets, LLM-as-judge), manage a small team, and deliver decision-ready insights to Product, Design, and Engineering stakeholders.
About the job
What you'll do at Figma
- Own AI evaluation methods and operations for Figma's AI-powered experiences — define quality dimensions, design how we measure them, and turn results into decision-ready signal
- Build and maintain evaluation frameworks, rubrics, golden datasets, and quality bars, combining human evaluation with automated/model-based approaches (e.g., LLM-as-judge) where appropriate
- Partner with engineering to stand up repeatable, reproducible evaluation pipelines and regression testing, so evaluation is a routine part of how AI features are built and shipped
- Produce clear readouts and dashboards that let stakeholders confidently make go/no-go and prioritization decisions
- Socialize a shared definition of quality so evaluation standards are adopted across teams rather than re-invented — and advocate for evaluation as a strategic partner in the product process
- Manage a small team to execute our AI evals in partnership with contractors, internal staff, and/or LLMs
Requirements
- 10+ years of experience in product, research, applied research, or a closely related field, including 2+ years of management experience
- Direct, hands-on experience owning the evaluation of AI/LLM-powered products
- Expertise designing and running AI evaluation — human evaluation programs, rubric and benchmark/golden-dataset construction, inter-rater reliability — and sound judgment about when and how to apply automated/model-based approaches (e.g., LLM-as-judge), including their limitations
- Strength across both qualitative and quantitative methods, comfort with data and metrics, and the ability to reason about model behavior
- Demonstrated success in identifying the riskiest assumptions behind an ambiguous quality question, prioritizing them, and designing right-sized evaluation to build confidence
- A proven track record of gaining buy-in from executive and cross-disciplinary stakeholders — transcending methodology to articulate a larger user story and the "so what" to inspire action
Nice-to-haves
- Experience building or co-building automated evaluation pipelines and regression testing in partnership with engineering, or familiarity with eval tooling (e.g., Braintrust, LangSmith, DeepEval, or equivalents)
- Experience standing up a new function, practice, or discipline from scratch
- 2+ years in product design, user-centric product management, data science, product development, and/or front-end engineering
- A familiarity and depth of experience using Figma's products
Skills
Ai Evaluation, Llm Evaluation, Human Evaluation, Rubric Design, Golden Datasets, Llm-As-Judge, Inter-Rater Reliability, Qualitative Methods, Quantitative Methods, Evaluation Pipelines, Regression Testing, Braintrust, Langsmith, Deepeval
Similar jobs
AI Research jobs
Ambral
New York, NYLeads the research agenda and hands-on development of replayable enterprise environments, agent evaluations, and post-training systems. The role requires deep AI research experience, a PhD or equivalent track record, and the ability to translate open-ended questions into production systems.
Deepgram
San Francisco, CALeads Deepgram’s end-to-end TTS research program, setting technical direction, training and evaluating large-scale speech-generation models, and turning breakthroughs into production systems. The role combines hands-on technical leadership with building and developing a high-performing research organization.
Order.co
Boston, MASets company-wide architecture and strategy for data and applied AI, connecting governed data foundations to production intelligence and measurable business outcomes. The role requires 14+ years of experience, strong production engineering judgment, executive partnership, and hands-on delivery.
Shield AI
San Mateo, CA
Leads design, implementation, integration, and field validation of tactical autonomy software for unmanned systems and multi-agent missions. Requires extensive autonomy or robotics experience, strong C++ and Python skills, and the ability to obtain a SECRET clearance.
Shield AI
Washington, DCLeads technical direction and develops maritime autonomy capabilities for unmanned surface and underwater vehicles, including motion planning, localization, safe behaviors, and heterogeneous multi-agent collaboration. Requires deep robotics and unmanned-systems experience, strong C++/Python skills, and senior technical leadership.