# Lead AI Product Engineer - AI Agents & Emerging Products
**Company:** [System1](https://hotfix.jobs/companies/system1)
**Location:** Los Angeles, CA
**Skills:** Python, Node.js, LLMs, Prompt Engineering, RAG, Vector Databases, AWS, GCP, Docker, CI/CD
**Posted:** 2026-01-14
> Lead the design, build, and operation of multi-agent AI systems from prototypes to production, owning architecture, reliability, and performance for consumer AI products. Requires strong production experience with LLM-powered systems, agentic workflows, and backend engineering.
## Job Description
## Responsibilities
- Build and operate multi-agent systems that plan, coordinate, and execute tasks across multiple tools and services
- Implement agentic coding workflows where agents generate code, run tests, create diffs or PRs, and iterate autonomously within guardrails
- Design agent orchestration layers (planners, supervisors, evaluators) to manage task decomposition, retries, and long-running execution
- Instrument and monitor agent behavior across success, failure, latency, cost, and human overrides
- Collaborate with Product & Design to translate user problems into shippable agent workflows

## Requirements
- Strong experience building LLM-powered systems in production, including prompt engineering, structured outputs, tool/function calling, and agentic coding workflows
- Solid backend engineering skills with Python and/or Node.js, designing APIs, async services, and background workers
- Experience designing multi-step, agent-driven execution loops with explicit state management, idempotency, and failure recovery
- Comfort with distributed systems primitives (queues, pub/sub, durable state, rate limiting, concurrency controls)
- Proven ability to instrument and observe agent behavior with structured events
- Experience deploying and operating systems on cloud infrastructure (AWS or GCP), using Docker and CI/CD pipelines
- Familiarity with inter-agent communication patterns (message schemas, shared memory, blackboard systems, or event-driven coordination)
- Experience implementing agent evaluation pipelines (prompt regression tests, simulation-based evals, automated scoring, human-in-the-loop review)
- Understanding of cost-aware execution for agents (dynamic model routing, caching, batching, budget constraints)
- Experience with RAG architectures, vector databases, and persistent memory strategies
- Exposure to emerging agent protocols (e.g., x402 payment flows, MCP)

## Benefits
- Competitive salary + bonus + equity
- Generous PTO + 11 company holidays
- Open sick time
- 100% covered Medical, Dental, Vision for employees
- 401k with match
- Health & Dependent Care Flex Spending Account
- Paid professional development
- Leadership & growth opportunities
- Virtual company and team building events
**Apply:** https://hotfix.jobs/jobs/lead-ai-product-engineer-ai-agents-emerging-products-at-system1-519cc89a-ed7c-4fde-86cc-45532e585ace
**Canonical:** https://hotfix.jobs/jobs/lead-ai-product-engineer-ai-agents-emerging-products-at-system1-519cc89a-ed7c-4fde-86cc-45532e585ace