# Lead AI Product Engineer - AI Agents & Emerging Products

**Company:** [System1](https://hotfix.jobs/companies/system1)
**Location:** Los Angeles, CA
**Role:** ML Engineering
**Skills:** Python, Node.js, LLMs, Prompt Engineering, RAG, Vector Databases, AWS, GCP, Docker, CI/CD
**Posted:** 2026-01-14

> Lead the design, build, and operation of multi-agent AI systems from prototypes to production, owning architecture, reliability, and performance for consumer AI products. Requires strong production experience with LLM-powered systems, agentic workflows, and backend engineering.

## Job Description

## Responsibilities
- Build and operate multi-agent systems that plan, coordinate, and execute tasks across multiple tools and services
- Implement agentic coding workflows where agents generate code, run tests, create diffs or PRs, and iterate autonomously within guardrails
- Design agent orchestration layers (planners, supervisors, evaluators) to manage task decomposition, retries, and long-running execution
- Instrument and monitor agent behavior across success, failure, latency, cost, and human overrides
- Collaborate with Product & Design to translate user problems into shippable agent workflows

## Requirements
- Strong experience building LLM-powered systems in production, including prompt engineering, structured outputs, tool/function calling, and agentic coding workflows
- Solid backend engineering skills with Python and/or Node.js, designing APIs, async services, and background workers
- Experience designing multi-step, agent-driven execution loops with explicit state management, idempotency, and failure recovery
- Comfort with distributed systems primitives (queues, pub/sub, durable state, rate limiting, concurrency controls)
- Proven ability to instrument and observe agent behavior with structured events
- Experience deploying and operating systems on cloud infrastructure (AWS or GCP), using Docker and CI/CD pipelines
- Familiarity with inter-agent communication patterns (message schemas, shared memory, blackboard systems, or event-driven coordination)
- Experience implementing agent evaluation pipelines (prompt regression tests, simulation-based evals, automated scoring, human-in-the-loop review)
- Understanding of cost-aware execution for agents (dynamic model routing, caching, batching, budget constraints)
- Experience with RAG architectures, vector databases, and persistent memory strategies
- Exposure to emerging agent protocols (e.g., x402 payment flows, MCP)

## Benefits
- Competitive salary + bonus + equity
- Generous PTO + 11 company holidays
- Open sick time
- 100% covered Medical, Dental, Vision for employees
- 401k with match
- Health & Dependent Care Flex Spending Account
- Paid professional development
- Leadership & growth opportunities
- Virtual company and team building events

## Similar roles

- [Senior Applied Scientist](https://hotfix.jobs/jobs/3ae25509-6ddb-4378-a140-0b121812bdbf) - DAT Freight & Analytics - Seattle, WA - $183k – $226k/yr
- [Senior Machine Learning Engineer, Relevance and Personalization](https://hotfix.jobs/jobs/589feee3-05a3-4944-84df-8bd20d220687) - Airbnb - Remote - $200k – $235k/yr
- [Senior Machine Learning Engineer, Agent Oversight](https://hotfix.jobs/jobs/1f504a69-044f-47ab-872b-120a1706cd88) - Scale AI - San Francisco, CA - $216k – $270k/yr
- [Senior Software Engineer, Agent Oversight](https://hotfix.jobs/jobs/808ce719-6b16-4b6e-8887-8c8f8a780b75) - Scale AI - San Francisco, CA - $216k – $270k/yr
- [Senior Software Engineer, AI Infrastructure](https://hotfix.jobs/jobs/dd1c3420-6c0a-47ed-a1a7-de401bdf506c) - Ambient.ai - Redwood City, CA - $168k – $205k/yr

**Apply:** https://hotfix.jobs/jobs/519cc89a-ed7c-4fde-86cc-45532e585ace
**Canonical:** https://hotfix.jobs/jobs/519cc89a-ed7c-4fde-86cc-45532e585ace