Skip to content
System1System1Los Angeles, CA

Lead AI Product Engineer - AI Agents & Emerging Products

Lead the design, build, and operation of multi-agent AI systems from prototypes to production, owning architecture, reliability, and performance for consumer AI products. Requires strong production experience with LLM-powered systems, agentic workflows, and backend engineering.

Salary not listed
On-siteML Engineering

About the role

Responsibilities

  • Build and operate multi-agent systems that plan, coordinate, and execute tasks across multiple tools and services
  • Implement agentic coding workflows where agents generate code, run tests, create diffs or PRs, and iterate autonomously within guardrails
  • Design agent orchestration layers (planners, supervisors, evaluators) to manage task decomposition, retries, and long-running execution
  • Instrument and monitor agent behavior across success, failure, latency, cost, and human overrides
  • Collaborate with Product & Design to translate user problems into shippable agent workflows

Requirements

  • Strong experience building LLM-powered systems in production, including prompt engineering, structured outputs, tool/function calling, and agentic coding workflows
  • Solid backend engineering skills with Python and/or Node.js, designing APIs, async services, and background workers
  • Experience designing multi-step, agent-driven execution loops with explicit state management, idempotency, and failure recovery
  • Comfort with distributed systems primitives (queues, pub/sub, durable state, rate limiting, concurrency controls)
  • Proven ability to instrument and observe agent behavior with structured events
  • Experience deploying and operating systems on cloud infrastructure (AWS or GCP), using Docker and CI/CD pipelines
  • Familiarity with inter-agent communication patterns (message schemas, shared memory, blackboard systems, or event-driven coordination)
  • Experience implementing agent evaluation pipelines (prompt regression tests, simulation-based evals, automated scoring, human-in-the-loop review)
  • Understanding of cost-aware execution for agents (dynamic model routing, caching, batching, budget constraints)
  • Experience with RAG architectures, vector databases, and persistent memory strategies
  • Exposure to emerging agent protocols (e.g., x402 payment flows, MCP)

Benefits

  • Competitive salary + bonus + equity
  • Generous PTO + 11 company holidays
  • Open sick time
  • 100% covered Medical, Dental, Vision for employees
  • 401k with match
  • Health & Dependent Care Flex Spending Account
  • Paid professional development
  • Leadership & growth opportunities
  • Virtual company and team building events

Skills

PythonNode.jsLLMsPrompt EngineeringRAGVector DatabasesAWSGCPDockerCI/CD

Similar roles

ML Engineering jobs
DAT Freight & Analytics

Senior Applied Scientist

DAT Freight & AnalyticsSeattle, WA

Senior Applied Scientist building and productionizing high-recall fraud detection, risk scoring, graph-based anomaly detection, and decision engines for Trust & Safety in a freight marketplace. Requires advanced degree, 5+ years production ML experience, and strong Python skills.

183k – 226k/yr
Hybrid5+ YOEML Engineering
Airbnb

Senior Machine Learning Engineer, Relevance and Personalization

AirbnbUnited States

Build and productionize cutting-edge ML models for Airbnb's query intelligence, including autocomplete, query tagging, expansion, intent modeling, and LLM-powered natural language search to understand guest intent.

200k – 235k/yr
Remote5+ YOEML Engineering
Scale AI

Senior Machine Learning Engineer, Agent Oversight

Scale AISan Francisco, CA +1

Senior ML Engineer building observability, evaluation frameworks, and improvement loops for production agentic AI systems. Requires 5+ years production ML/LLM experience, strong grounding in agent design or evaluation, and hands-on work taking systems from prototype to scale.

216k – 270k/yr
On-site5+ YOEML Engineering
Scale AI

Senior Software Engineer, Agent Oversight

Scale AISan Francisco, CA +1

Build platform infrastructure for observing, evaluating, and improving production AI agents at scale. Requires 4+ years software engineering experience with ML/LLM systems, strong backend/distributed systems skills, and collaboration with ML engineers.

216k – 270k/yr
On-site5+ YOEML Engineering
Ambient.ai

Senior Software Engineer, AI Infrastructure

Ambient.aiRedwood City, CA

Build and optimize scalable AI infrastructure for real-time inference, evaluation, and continuous improvement of LLMs, LVMs, computer vision, and multimodal models on large-scale video data. Requires 4+ years production ML systems experience, strong Python skills, and expertise in inference optimization and model serving.

168k – 205k/yr
Hybrid4+ YOEML Engineering