# Model Behavior Architect

**Company:** [Perplexity](https://hotfix.jobs/companies/perplexity)
**Location:** San Francisco, CA
**Role:** ML Engineering
**Salary:** $180k – $260k/yr
**Skills:** Python, LLMs, Prompt Engineering, Context Engineering, Ai Evaluations, Benchmarks, A/B Testing, Perplexity Ai
**Posted:** 2026-01-15

> Designs prompt and context engineering strategies for AI products, builds evaluation pipelines, and ensures model quality and behavioral consistency. Requires 3+ years with LLMs, Python experience, and expertise in AI evaluations.

## Job Description

## Key Responsibilities
- **Context Engineering:** Design, test, and optimize context strategies and system prompts that shape answer engine behavior across products, features, and use cases.
- **Evaluation Systems:** Build automated and semi-automated evaluation pipelines that measure model quality, catch regressions, and scale across product surfaces.
- **Model Launch Support:** Partner with research and engineering to validate model behavior before and during rollouts, ensuring smooth transitions with no degradation.
- **Research & Analysis:** Identify inconsistencies and failure modes in model outputs through well-designed research projects — for both internal and production-facing systems.
- **Cross-functional Collaboration:** Work closely with design, product, and research teams to translate product goals into concrete model behavior requirements.
- **Knowledge Sharing:** Help engineers across teams build intuition for prompt design, context engineering, and evaluation best practices.
- **Staying Current:** Track the latest alignment, evaluation, and prompting techniques from industry and academia, and bring the best ideas back to the team.

## What We're Looking For

### Required
- Experience designing evaluations, benchmarks, or metrics for AI systems.
- Strong written and verbal communication skills, particularly in explaining complex concepts to diverse stakeholders.
- Ability to manage multiple concurrent projects in a fast-moving environment.
- Strong experience with Perplexity or other frontier AI models in production settings.
- Demonstrated experience with Python — you'll prototype, debug, automate, and build systems at scale.
- 3+ years of experience working with LLMs in a product or research setting.

### Preferred
- Experience with A/B testing or experimentation frameworks.
- Track record of improving AI system performance through systematic evaluation and iteration.

## Similar roles

- [Software Engineer, AI Platform](https://hotfix.jobs/jobs/7ddad000-3e1c-47cb-a45b-31d8e72abc4f) - Notion - San Francisco, CA - $180k – $201k/yr
- [Research Engineer, Generalist](https://hotfix.jobs/jobs/1308107a-d911-4d6a-924d-e87d4654d356) - Exa - San Francisco, CA - $180k – $350k/yr
- [Software Engineer - BIS](https://hotfix.jobs/jobs/e9bc193b-be22-4ba7-8730-9d1ea41c31ef) - Baseten - San Francisco, CA - $180k – $360k/yr
- [AI Engineer (Core)](https://hotfix.jobs/jobs/36b94901-fb9d-47d4-a46b-51ef8679ea67) - Build - San Francisco, CA - $180k – $250k/yr
- [Research Engineer](https://hotfix.jobs/jobs/54931695-a06a-4671-b30e-cacd5daf8c92) - Lightning AI - Remote - $180k – $250k/yr

**Apply:** https://hotfix.jobs/jobs/59710409-28d3-4a11-89b8-e92865d1c328
**Canonical:** https://hotfix.jobs/jobs/59710409-28d3-4a11-89b8-e92865d1c328