Senior/Staff - Agent Platform
Build and scale evaluation, experimentation, and quality systems for Cortex Code’s enterprise coding agents. The role requires 10+ years shipping AI/ML-backed production software, strong programming skills, and Staff-level technical leadership across engineering, modeling, and product.
About the job
Responsibilities
- Own major pillars of the quality stack, tuning agent behavior for next-generation agentic coding tasks.
- Design and evolve pipelines and tooling for large-scale experimentation, error mining, and iteration on prompts, tools, and workflows with clear before-and-after signals.
- Lead postmortems on quality regressions, cluster failure modes, and translate findings into a prioritized roadmap for engineering and modeling partners.
- Align product, infrastructure, and applied AI teams on quality standards for critical customer workflows.
- Mentor engineers and improve evaluation practices across the team.
- Ensure quality systems are dependable through reproducible runs, stable datasets, versioning, and operational clarity when systems drift.
Requirements
- Bachelor's degree in Computer Science, Engineering, Statistics, or a related field.
- 10+ years of experience shipping AI/ML-backed software in production, including Staff-level ownership of technical direction, cross-team delivery, and mentoring.
- Strong track record building and operating evaluation harnesses, measurement systems, and/or experimentation loops for LLM or agent systems.
- Proficiency in programming languages such as Python, TypeScript, and Go; strong in at least two.
- Exceptional communication skills, including crisp technical writing, constructive debate, and influencing without authority across engineering and product.
Nice to Have
- Experience with data engineering pipelines, data modeling, data analysis, retrieval systems, and semantic layers.
- Deep experience with agentic coding tools, including IDE and CLI agents, and intuition for model strengths, failure modes, and prompting limits.
- Background in data engineering, analytics, retrieval-augmented generation, or semantic layers.
- Experience with LLM observability, safety and guardrails, or quality systems used as production release gates.
- Experience building and owning complex quality and data pipelines with substantial state, branching logic, and operational requirements.
Compensation
- Annual salary range: $236,000–$309,750.
Skills
Python, TypeScript, Go, LLMs, AI/ML, Agentic Systems, Evaluation Harnesses, Experimentation, Data Engineering, dbt, Airflow, Retrieval-Augmented Generation, Llm Observability, Semantic Layers, Prompt Engineering
Similar jobs
ML Engineering jobs
Snowflake
Bellevue, WA
Leads the roadmap and technical vision for Snowflake Feature Store, building reliable, high-performance machine learning platform capabilities and supporting technical execution across partner teams. Requires 10+ years of experience with data-serving infrastructure or ML platforms, plus Java and Python expertise.
Shield AI
San Mateo, CA
Leads the design, implementation, integration, and field validation of tactical autonomy and multi-agent coordination capabilities for unmanned platforms. Requires 7+ years of relevant experience, production C++, technical leadership, and eligibility for a U.S. Secret clearance.
Harvey
San Francisco, CA
Leads the design and operation of reliable, scalable model infrastructure powering AI inference across multiple providers. Requires 7+ years of distributed-systems engineering experience, strong programming skills, and expertise in production reliability and cloud infrastructure.
Airbnb
United States
Senior Staff ML Engineer fine-tunes and optimizes state-of-the-art LLMs for Airbnb's customer support AI products, including AI assistants and autonomous agents. Partners cross-functionally to productionize models at scale. Requires PhD and 10+ years experience with PyTorch.
Haus
Seattle, WAStaff Machine Learning Engineer building and operating production ML systems for causal marketing measurement, optimization, and planning. The role requires deep statistical and machine learning expertise, production programming experience, cross-functional collaboration, and technical mentorship.