# Senior Software Engineer, Engine & Distributed Systems

**Company:** [StackAI](https://hotfix.jobs/companies/stackai)
**Location:** San Francisco, CA, New York, NY
**Role:** Backend Engineering
**Salary:** $202k – $230k/yr
**Experience:** 5+ years
**Skills:** Python, FastAPI, Postgres, Temporal, cadence, Distributed Systems, Concurrency, queueing, idempotency, state machines, failure recovery, event-driven architectures, message queues, LangGraph, Pydantic
**Posted:** 2026-06-19

> Senior Software Engineer owning the durable execution engine and distributed runtime for long-running AI agents and workflows at StackAI. Requires deep expertise in distributed systems, workflow orchestration (Temporal/Cadence), Python backends, concurrency, fault tolerance, and building reliable checkpointing/recovery at scale.

## Job Description

## What you'll do
- Own the execution engine: the runtime, scheduling, and sub-agent parallelization that run every agent on the platform.
- Make long-running work durable: build checkpointing, resumption, and recovery so agents survive failures and restarts and pick up exactly where they left off.
- Shape the execution model: decide how work is scheduled, queued, and moved from synchronous to asynchronous, so the platform stays correct and responsive as load grows.
- Engineer for scale and reliability: hold the engine to strict health targets for worker freshness, deploy safety, and drain time, and keep latency and throughput strong as volume grows.
- Keep the engine open to the ecosystem: make it straightforward to bring new agent harnesses, orchestration frameworks, and model capabilities into the runtime.

## Requirements
- 5+ years building backend systems in production, with real depth in distributed systems.
- Hands-on experience with durable execution or workflow orchestration (Temporal, Cadence, or equivalent), with a way of thinking rooted in idempotency, state machines, and failure recovery.
- Strong command of concurrency, queueing, retries, and fault tolerance under load.
- Strong in Python and modern backend frameworks (FastAPI or similar), with sound database fundamentals (Postgres or similar).
- Drawn to the correctness problems that everything else quietly depends on.

## Nice-to-haves
- Operating Temporal at scale.
- Event-driven architectures and message queues.
- Experience with Pydantic, AI, LangGraph, or similar.
- AI or agent runtimes: tool-calling, sub-agent orchestration, streaming.
- Performance and cost optimization of high-throughput backends.
- Startup or growth-stage experience.

## Similar roles

- [Senior Software Engineer, Identity & Access](https://hotfix.jobs/jobs/530df964-d55f-4211-9e61-f575fee58e3c) - StackAI - San Francisco, CA - $202k – $230k/yr
- [Senior Software Engineer](https://hotfix.jobs/jobs/1ab397f5-5435-4dfd-af3b-9c726c8407cd) - Mercury - Remote - $201k – $251k/yr
- [Senior Software Engineer, Streaming](https://hotfix.jobs/jobs/c74f3312-5241-4d70-85ff-1cefce151793) - Skydio - San Mateo, CA - $200k – $240k/yr
- [Postgres Product Engineer](https://hotfix.jobs/jobs/383d1f82-2b1d-4f73-8b74-475640a0429b) - Render - Remote - $204k – $280k/yr
- [Software Engineer, Backend](https://hotfix.jobs/jobs/542f1aa3-f562-44e7-a8de-caf3d127868d) - Mercor - San Francisco, CA - $200k – $500k/yr

**Apply:** https://hotfix.jobs/jobs/142fc0de-9006-4f99-a6bc-8c50d3741d75
**Canonical:** https://hotfix.jobs/jobs/142fc0de-9006-4f99-a6bc-8c50d3741d75