# Staff Software Engineer, Agent Orchestration

**Company:** [Decagon](https://hotfix.jobs/companies/decagon)
**Location:** San Francisco, CA
**Role:** Backend Engineering
**Salary:** $200k – $400k/yr
**Experience:** 7+ years
**Skills:** Distributed Systems, Backend Platforms, Agent Harnesses, Model Orchestration, Control Planes, A/B Testing, Offline Evaluation, Observability, Voice Systems, Transcription Pipelines
**Posted:** 2025-12-01

> Design and operate the distributed runtime, orchestration, control-plane, and experimentation systems that govern production AI agents. The role requires staff-level experience building reliable backend platforms, debugging complex systems, and improving latency, safety, observability, and execution quality.

## Job Description

## Responsibilities
- Design and evolve agent harnesses powering different product experiences.
- Build core runtime systems, including AOP execution and multi-model orchestration.
- Develop control-plane logic for routing, planning, and tool invocation with strong safety guarantees.
- Optimize agent systems for latency, reliability, and production correctness.
- Analyze real-world failures and use data to drive iterative improvements.
- Build and operate online experimentation (A/B testing) and contribute to offline evaluation frameworks.
- Improve observability, testing, and simulation systems to ensure safe, measurable progress.
- Contribute to voice and real-time systems, including transcription pipelines, turn-taking, and latency improvements.
- Continuously adapt orchestration systems as model capabilities evolve.

## Requirements
- Strong experience building distributed systems or backend platforms in production environments.
- Comfort working in ambiguous, fast-moving environments with rapid iteration cycles.
- Experience owning systems end-to-end, from design through production and iteration.
- Familiarity with experimentation, evaluation, or data-driven product improvement loops.
- Track record of improving system reliability, performance, and observability.
- Ability to debug complex systems and identify root causes of failures.

## Nice-to-haves
- Experience building or working on agent harnesses, orchestration layers, or execution frameworks.
- Systems-level thinking around control planes, feedback loops, and optimization.
- Interest in diagnosing failure modes and iterating toward measurable improvements.
- Focus on production quality, reliability, safety, and scalability.
- Motivation to advance intelligent systems in real-world environments.

## Compensation and Benefits
- Base salary: $200K–$400K, plus equity.
- Medical, dental, and vision benefits for employees and families.
- Life insurance and disability benefits.
- Retirement plan.
- Parental leave.
- Fertility and family-building benefits through Carrot.
- Monthly wellness and lifestyle stipend.
- Daily office lunches and snacks.
- Take-what-you-need vacation policy.

## Similar jobs

- [Staff Engineer, Safety-Critical Software](https://hotfix.jobs/jobs/7325e489-d1bd-4822-ac88-77470fa68c78) - Shield AI - San Diego, CA - $200k – $300k/yr
- [Staff Software Engineer - Detection Platform](https://hotfix.jobs/jobs/4a4b435d-52a0-4439-be2a-2c9bdd63448b) - Huntress - Remote - $200k – $220k/yr
- [Senior / Staff Backend Engineer, Maps](https://hotfix.jobs/jobs/d259d606-ad53-4f4b-97e1-013a5cdd068c) - Radar Labs - New York, NY - $200k – $300k/yr
- [Senior Staff Software Engineer](https://hotfix.jobs/jobs/3f388646-0ba6-4256-bce9-76ba7aecfede) - Webflow - Remote - $205k – $316k/yr
- [Staff Engineer - Cloud Security](https://hotfix.jobs/jobs/b7355bc4-a593-4a0c-8b75-a29e67b18f27) - Illumio - Sunnyvale, CA - $194k – $233k/yr

**Apply:** https://hotfix.jobs/jobs/287aa70c-34f3-4364-98cd-80b5b82be8aa
**Canonical:** https://hotfix.jobs/jobs/287aa70c-34f3-4364-98cd-80b5b82be8aa