# Staff Software Engineer, AI Platform

**Company:** [Watershed](https://hotfix.jobs/companies/watershed)
**Location:** San Francisco, CA
**Role:** ML Engineering
**Salary:** $202k – $255k/yr
**Experience:** 6+ years
**Skills:** TypeScript, LLMs, Embeddings, Machine Learning, Agent Infrastructure, Observability, Model Evaluation, Backend Engineering, Production Systems, Ai/Ml Engineering
**Posted:** 2026-06-16

> Technical leader building agent infrastructure, observability, evals, and guardrails for production AI systems at Watershed. Requires 6+ years backend/platform/AI engineering experience and production TypeScript systems.

## Job Description

## Responsibilities
- Design and build the agent infrastructure that powers Watershed's products
- Develop the observability and tracing layer for agent decisions, making it possible to debug, evaluate, and improve agent behavior at scale
- Build evals, harnesses, and guardrails that turn agent capabilities into production-grade, dependable systems
- Collaborate with product and other AI engineering teams to set product and technical strategy, and define the boundaries between autonomous agent behavior, deterministic code, and human oversight
- Keep up with developments and state-of-the-art in AI and agent infrastructure to determine what is relevant to Watershed
- Work closely with Watershed product teams to contribute your expertise to build agent experiences across the product
- Write performant, well-crafted, tested, and maintainable code across our technical stack

## Requirements
- 6+ years of experience in backend, platform, or AI/ML engineering
- Experience building products and infrastructure that leverage LLMs, embeddings, and other ML technologies
- Full lifecycle experience building, deploying, and monitoring production systems that depend on LLMs or other ML technologies
- Experience with model evaluation, agent observability, and making non-deterministic systems reliable
- Experience building and operating production Typescript systems
- Must be willing to work from an office 4 days per week

## Similar roles

- [Staff AI Engineer](https://hotfix.jobs/jobs/c1f04a88-e965-40fd-a5e3-f14209540707) - Drata - Remote - $201k – $272k/yr
- [Senior / Staff AI Platform Engineer](https://hotfix.jobs/jobs/65f2c893-cd67-482a-9cfb-623b499605ee) - Clear Street - Remote - $200k – $350k/yr
- [Senior / Staff AI Model Engineer](https://hotfix.jobs/jobs/efc2964d-f187-4b13-b8a3-2e78080490a2) - Clear Street - Remote - $200k – $350k/yr
- [Staff Software Engineer, Agents](https://hotfix.jobs/jobs/353f3eaf-bf6e-4405-92aa-c39af858b5ac) - Decagon - San Francisco, CA - $200k – $400k/yr
- [Member of Technical Staff - Research Fellow](https://hotfix.jobs/jobs/319ee852-b158-4ee4-b29a-a680b8bfcfba) - Nuance Labs - Seattle, WA - $200k – $250k/yr

**Apply:** https://hotfix.jobs/jobs/fac2c8c7-4aea-4cfc-8698-a37c842a23ab
**Canonical:** https://hotfix.jobs/jobs/fac2c8c7-4aea-4cfc-8698-a37c842a23ab