# Software Engineer, AI Agent & LLM

**Company:** [Otter](https://hotfix.jobs/companies/otter)
**Location:** Mountain View, CA
**Role:** ML Engineering
**Salary:** $155k – $185k/yr
**Experience:** 3+ years
**Skills:** AI Agents, LLMs, Machine Learning, backend engineering, Distributed Systems, evaluation systems, Prompt Engineering, Data Pipelines, Observability, Python
**Posted:** 2026-07-22

> Build and improve AI agent experiences and conversational knowledge engine for Otter AI Chat. Focus on quality evaluation, infrastructure for orchestration/tracing, diagnosing failures across the stack, and driving measurable improvements in AI systems from traces and feedback. Requires 3+ years AI/ML engineering, strong backend/distributed systems skills, and experience shipping LLM-powered products.

## Job Description

## Responsibilities
- Build AI-agent experiences for Otter AI Chat that help users reason over conversations, retrieve knowledge, and complete complex tasks.
- Advance Otter’s conversational knowledge engine through better knowledge extraction, annotation, and indexing.
- Develop robust evaluation datasets, automated graders, regression tests, and release gates for AI quality.
- Diagnose failures across models, prompts, retrieval, tools, data pipelines, backend services, and product workflows.
- Build shared agent infrastructure for orchestration, tracing, debugging, retries, sandboxed execution, and observability.
- Improve task completion, correctness, groundedness, reliability, latency, and cost.
- Turn production traces, customer feedback, and usage signals into measurable product and model improvements.
- Collaborate across AI, product, infrastructure, data, security, and application teams to deliver end-to-end capabilities.

## Requirements
- 3+ years of AI Agent engineering, machine learning engineering, or related experience.
- Strong backend or distributed-systems engineering skills.
- Hands-on experience building and shipping AI-agent or LLM-powered products.
- Strong focus on AI quality and experience evaluating nondeterministic systems.
- Ability to use data, traces, logs, and qualitative examples to identify and resolve complex failures.
- Works effectively across services, technical domains, and organizational boundaries.
- Combines strong product judgment with rigorous engineering and evaluation practices.
- High agency, strong ownership, and a bias toward action.
- Ability to take ambiguous problems from initial exploration through production launch and continuous improvement.

## Nice-to-Haves
- Demonstrated ability to use coding agents effectively while rigorously reviewing and controlling the quality of their output.

## Similar roles

- [Applied AI Engineer](https://hotfix.jobs/jobs/595fc12f-b7aa-4e1a-a838-2af92d0ed5bd) - Ramp - New York, NY - $155k – $340k/yr
- [Machine Learning Engineer](https://hotfix.jobs/jobs/d7058f3f-7306-4fa8-8526-ed2d9efe83c9) - Twilio - Remote - $156k – $229k/yr
- [Software Engineer - E2E Autonomy](https://hotfix.jobs/jobs/e3b6ebe4-5501-421b-ae1a-a736896881af) - Applied Intuition - Sunnyvale, CA - $153k – $222k/yr
- [AI engineer](https://hotfix.jobs/jobs/cbe50e1e-cfc8-40e3-82dc-cb84345430d6) - Writer - San Francisco, CA - $152k – $316k/yr
- [Software Engineer, Forward Deployed Agent Builder](https://hotfix.jobs/jobs/ae0bb066-9d72-4c0e-b443-4a6366ab595d) - Brex - Seattle, WA - $152k – $240k/yr

**Apply:** https://hotfix.jobs/jobs/93c308f0-3f0d-40a8-b807-9c9242fb1d6c
**Canonical:** https://hotfix.jobs/jobs/93c308f0-3f0d-40a8-b807-9c9242fb1d6c