Product Designer, Evals & Prompts
Build LLM product evaluations, prompt improvements, and designer-facing tools that measure and improve Claude’s behavior across product surfaces and model releases. The role requires production Python, evaluation-pipeline experience, internal tool development, and prompt or model-launch experience.
About the job
Responsibilities
- Write, test, revise, and ship prompts for Claude’s tools, features, and product behaviors; verify that shipped prompts match the intended versions.
- Build graders, automated evaluations, comparison sets, regression suites, and the infrastructure that runs them across models.
- Develop visual, low-code evaluation tools that enable designers to assemble transcript sets, create graders from plain-English rubrics, compare prompt variants, and inspect results.
- Observe designers using evaluation tools and simplify the workflows.
- Support model releases by testing product surfaces, creating prompt fixes and migrations, and developing prompts for new features.
- Build and scale a reliable evaluation harness for tool calls, including sandboxing, consistent settings, and regression diagnosis.
- Package behaviors for training using graders, human-feedback questions, and preference pairs.
Requirements
- Production-quality Python.
- Experience building and maintaining evaluation pipelines for LLM products, including graders, rubrics, comparison sets, regression suites, and model-run infrastructure.
- Experience building internal tools with interfaces for nontechnical users.
- Experience standing up test harnesses, sandboxing tool calls, and pinning comparable run settings.
- Experience shipping prompts or collaborating closely with prompt engineers, with an understanding of cross-model prompt behavior.
- Ability to analyze transcripts in addition to evaluation scores.
- Bachelor’s degree or equivalent combination of education, training, and experience in a relevant field.
Nice-to-haves
- Experience working within model-launch cycles.
- A/B testing experience and ability to connect offline evaluations to online outcomes.
- Front-end development or notebook-to-application experience, with strong opinions about evaluation-result usability.
- Experience turning product rubrics into training signals, including graders, human-feedback questions, or preference pairs.
- Product-focused approach to model behavior and user experience.
Compensation
- Annual salary: $305,000–$385,000 USD.
Skills
Python, Llm Evaluations, Prompt Engineering, Evaluation Pipelines, Graders, Regression Testing, Test Harnesses, Sandboxing, A/B Testing, Front-End Development, Human Feedback, Preference Pairs
Similar jobs
Product Design jobs
OpenAI
San Francisco, CA
Leads a product design team and remains hands-on in shaping growth experiences for Codex and ChatGPT business users. The role requires strong product judgment, interaction and visual craft, mentorship, experimentation, and the ability to translate complex AI capabilities into clear user experiences.
OpenAI
San Francisco, CA
Leads product design for identity, security, permissions, and administration across consumer and enterprise products. The role requires strong systems thinking, UX/UI and interaction design, and experience translating complex policies and access models into clear, trustworthy workflows.
Baseten
San Francisco, CAProduct Designer responsible for end-to-end design of a technical AI platform, partnering closely with engineers, product teams, and customers. The role requires strong visual and interaction design, product judgment, prototyping skills, and comfort with developer tools and front-end technologies.
F2
San Francisco, CA
Product Designer responsible for shaping end-to-end experiences across a complex B2B AI platform for private markets. The role requires 5+ years of product design experience, strong visual and interaction design skills, and effective collaboration with product and engineering teams.
OpenAI
New York, NY
Design and ship high-quality, responsive web experiences for business audiences, translating complex AI products and use cases into clear, compelling journeys. The role requires strong visual craft, UX thinking, systems design, and close collaboration with engineering and marketing.