AI Engineer, Model Quality and Performance
Own model quality and performance for Cerebras inference by building AI agent-driven eval suites, automating benchmarking, and creating customer-specific tooling. Requires strong AI agent experience and tooling intuition.
About the job
What You'll Do
- Design eval suites with AI agents in the loop. For every model release, curate a thoughtful mix of advanced, basic, long-context, and customer-use-case-specific evals. Use Claude to generate, validate, and prune candidate test cases at speed.
- Build custom evals for target customers by orchestrating AI agents to mine trajectories from their workloads and synthesize representative eval sets.
- Automate eval execution end-to-end with AI-driven pipelines on top of standard tooling (Docker, Git, CI). The goal is a system that runs itself between releases, not a script you re-run by hand.
- Build automations to forecast and benchmark model performance on Cerebras for our top customers, including modeling how fast customer-specific workloads will run in production.
- Build product-quality tooling that synthesizes quality + performance data into a single, easy-to-use view.
Skills & Qualifications
- Experience building AI agents. You ship real systems with Claude (or equivalent) as a force multiplier. You've built things that would have been infeasible solo without AI agents in the loop.
- Strong math/stats background.
- Comfort with Docker, Git, and the standard automation stack.
- A taste for tooling design. You've shipped something that a non-engineer used without complaining. Bonus if AI helped you ship it.
Assets
- Performance-tuning experience on custom silicon, GPUs, or FPGAs.
- Experience designing evals for agentic / coding / long-context / multimodal use cases.
- Familiarity with open-source eval frameworks (EvalScope, lm-eval-harness, etc.).
Skills
AI Agents, Claude, Docker, Git, CI/CD, Model Evaluation, Performance Benchmarking, Python, Math/Stats, Lm-Eval-Harness
Similar jobs
ML Engineering jobs
Anthropic
San Francisco, CABuild and optimize a high-scale LLM inference engine spanning accelerator programming, host-device coordination, and distributed systems. The role requires strong systems programming, performance analysis, and an understanding of LLM inference across compute, memory, and interconnects.
OnePay
United States
Build and operate production AI agents, automation workflows, and integrations that improve complex business processes. The role requires 5+ years of software engineering experience, modern LLM and agent-framework expertise, systems integration skills, and strong cross-functional collaboration.
Thinking Machines Lab
San Francisco, CA
Build and operate the engineering systems that support post-training research, including reinforcement learning infrastructure, sandboxed execution, data pipelines, and agent scaffolding. The role requires strong Python and systems engineering skills, project ownership, and a relevant bachelor’s degree or equivalent experience.
OpenAI
San Francisco, CA
Build research infrastructure and tooling that enables AI models to design silicon, including reinforcement learning environments, EDA integrations, evaluations, and experiment workflows. The role requires strong software engineering fundamentals and comfort working across research, tooling, and chip-design systems.
Rollstack
United StatesBuild production AI capabilities for automated slide and document generation, working across LLM applications, data analysis, and content generation. The role requires 3+ years in machine learning and NLP, advanced Python, and experience with LLM frameworks and production systems.